Nelson was born in Fort Payne, Alabama and grew up in Acworth, Georgia, northwest of Atlanta. He didn't play the game as a child – atypical for a successful professional golfer – in high school he focused on basketball and baseball. Nelson took up golf at the age of 21 after he returned from serving in the infantry in Vietnam (Nelson was a 20-year-old newlywed when he was drafted into the U.S. Army). Nelson was first introduced to golf by Ken Hummel, a soldier and friend in his infantry unit, and Nelson carefully studied Ben Hogan's book The Five Fundamentals of Golf while learning how to play the game. He soon found that he had a talent for the game, breaking 100 the first time he played and 70 within nine months. He went on to graduate from Kennesaw Junior College in 1970 and turned professional the following year. He qualified for the PGA Tour at 27. His breakthrough year came in 1979, when he won twice and finished second on the money list to Tom Watson.
Nelson won 10 times on the PGA Tour, including three major championships. He earned his first major title at the 1981 PGA Championship, which he won by four strokes. In 1983 he was victorious at the U.S. Open at Oakmont, coming from seven behind at the half-way point to defeat Tom Watson by a single shot. Nelson scored a U.S. Open record 65-67 over the last 36 holes at the difficult Oakmont course, which broke a 51 year Open record established by Gene Sarazen. Nelson's 10 under par 132 record score has yet to be equaled. In 1987 he finished tied with Lanny Wadkins after the regulation 72 holes of PGA Championship, and won the title with a par at the first playoff hole.
Nelson played on the U.S. Ryder Cup team in 1979, 1981, and 1987. His record of 9–3–1 is one of the best since the event became USA v Europe in 1979; it had been a perfect 9–0–0 after the first two events. He also won four tournaments on the Japan Golf Tour.
Since turning 50 in 1997 Nelson has had a successful Champions Tour career, winning 19 times, although he has not won a senior major. He was elected to the World Golf Hall of Fame in April 2006 and inducted in October 2006. In 2011, Nelson received the PGA Distinguished Service Award from the PGA of America. The award "honors outstanding individuals who display leadership and humanitarian qualities, including integrity, sportsmanship and enthusiasm for the game of golf".