Richardson was born in Spring Hope, North Carolina. After completing high school in Fayetteville, North Carolina, he entered Wofford College. Richardson was an Associated Press Little All-America selection in 1957 and '58. He still holds Wofford's single-game record with 241 receiving yards vs. Newberry in 1956 and is the record holder for touchdown receptions in a season (9 in 1958) and in a career (21). As a senior at Wofford, he scored 72 points on nine touchdowns, 12 extra points and two field goals. Richardson calls his greatest honor being elected team captain in 1958. In 1983, he was chosen to Wofford's All-Time Football team as a receiver.
Richardson was also active in numerous groups on the Wofford campus. He was a member of Kappa Alpha Order fraternity, President of the Inter-Fraternity Council, and member of the SCA Cabinet. Honors he received while at Wofford included Distinguished Military Student, Scabbard and Blade Military Fraternity, Sigma Delta Psi, Blue Key National Honorary Fraternity, and recognition in "Who's Who in American Universities and Colleges".
Drafted in the 13th round by the defending world champion Colts, Richardson played two seasons in the NFL, earning Colt Rookie of the Year honors in 1959. He caught a touchdown pass in the 1959 NFL Championship Game from quarterback Johnny Unitas.
Following his NFL career, Richardson embarked on a successful business career. Richardson used his 1959 NFL championship bonus with the help of Charles Bradshaw to open the first Hardee's Franchise in Spartanburg, South Carolina. The two ended up owning the Hardee's business 50/50. The business expanded rapidly under his hands-on management style. From headquarters in Spartanburg, he co-founded Spartan Foods, which was the first franchisee of Hardee's. He later was the CEO of Flagstar, which was the sixth largest food service company in the nation, controlling 2,500 restaurants and 100,000 employees, and retired in 1995.
On October 26, 1993, Richardson became the first former NFL player since George Halas to become an owner when the Carolina Panthers were unanimously awarded the NFL's 29th franchise.
Richardson has played a prominent role locking out the NFL players in 2011 and in negotiating a new players agreement. In 2015 he was elected to the South Carolina Sports Hall of Fame.
It had long been presumed that Richardson intended to have his sons, Mark and the late Jon (who died in July 2013 from cancer), inherit the team. However, both stepped aside before the 2009 season. On January 16, 2013, WBTV in Charlotte reported that Richardson wants the team sold after he dies, but presumably only to someone who will keep the team in Charlotte.
Richardson was hospitalized in Charlotte at Carolinas Medical Center in early December 2008, one month after receiving a pacemaker. Richardson, who had a history of heart trouble and had undergone quadruple bypass surgery in 2002, was placed on a donor waiting list for a new heart two days later. He received a new heart on February 1, 2009, and he has recovered from the transplant.