Jones grew up in Ebbw Vale, Wales and ran his first race at the age of 15 as a member of the Air Training Corps. Dissatisfied with working in a sewing-machine shop, he became an aircraft technician for the Royal Air Force (RAF) in 1974, and joined the RAF's running team in 1976. After receiving an invitation to the 1983 Chicago Marathon Jones began training for that distance; he previously had specialized in the 5,000 and 10,000 m, and finished 8th in the 1984 Olympic 10,000 m event,
In 1984, a year after dropping out of the Chicago Marathon because of injury, Jones won the event—his first completed marathon—with a time of 2:08:05, breaking the world record of Australian Robert de Castella by 13 seconds. Jones was unaware of the record and, since he never wore a watch, did not know that he might break it until two miles before the finish. He won the 1985 London Marathon in 2:08:16 despite stopping to go to the bathroom during the event; Jones later said, "I didn't even train for [the race]", instead continuing to coach himself. Aware that he was "one hamstring tear away from oblivion", Jones remained with the RAF despite earning large sums per race. In August of 1985, he broke the world record for the half marathon, running 61:14 in Birmingham. On October 20th of that year he achieved his personal best marathon time of 2:07:13 in winning the Chicago Marathon, only one second slower than the world record run by Carlos Lopes at the Rotterdam Marathon earlier that same year. This time remains the fastest of any British runner.
In 1986, he won a bronze medal in the 10,000 m at the Commonwealth Games. In the European Championships shortly after, he once again competed in the marathon. Leading from the start and breaking away from the pack, Jones ran a brilliant race up to the 20 mile mark. At that point he was leading by over two minutes and on schedule for another world record. However, he then "hit the wall" and suffered terribly in the final six miles. He slowed to a virtual shuffle, but refused to quit as he watched other competitors catch and pass him. Two years later, in the 1988 New York City Marathon, Jones won by over three minutes with a time of 2:08:20.
He was the first Welsh athlete to appear on the cover of the prestigious running magazine Running Times.