Richardson was born in Spring Hope, North Carolina. After completing high school in Fayetteville, North Carolina, he entered Wofford College in Spartanburg, South Carolina. Richardson was an Associated Press Little All-America selection in 1957 and '58. He still holds Wofford's single-game record with 241 receiving yards vs. Newberry in 1956 and is the record holder for touchdown receptions in a season (9 in 1958) and in a career (21). As a senior at Wofford, he scored 72 points on nine touchdowns, 12 extra points and two field goals. Richardson calls his greatest honor being elected team captain in 1958. In 1983, he was chosen to Wofford's All-Time Football team as a receiver.
Richardson was also active in numerous groups on the Wofford campus. He was a member of Kappa Alpha Order fraternity, President of the Inter-Fraternity Council, and member of the SCA Cabinet. Honors he received while at Wofford included Distinguished Military Student, Scabbard and Blade Military Fraternity, Sigma Delta Psi, Blue Key National Honorary Fraternity, and recognition in "Who's Who in American Universities and Colleges". [1a]
Drafted in the 13th round by the defending world champion Colts, Richardson played two seasons in the NFL, earning Colt Rookie of the Year honors in 1959. He caught a touchdown pass in the 1959 NFL Championship Game from quarterback Johnny Unitas.
Following his NFL career, Richardson embarked on a successful business career. Richardson used his 1959 NFL championship bonus with the help of Charles Bradshaw to open the first Hardee's franchise in Spartanburg. The two ended up owning the Hardee's business 50/50. The business expanded rapidly under his hands-on management style. From headquarters in Spartanburg, he co-founded Spartan Foods, which was the first franchisee of Hardee's. He later was the CEO of Flagstar, which was the sixth largest food service company in the nation, controlling 2,500 restaurants and 100,000 employees, and retired in 1995.
On October 26, 1993, Richardson became the first former NFL player since George Halas to become an owner when the Carolina Panthers were unanimously awarded the NFL's 29th franchise.
For the most part, Richardson has largely stayed in the background and rarely interferes in the Panthers' day-to-day operations. For instance, when he fired George Seifert after the 2001 season (in which the Panthers went 1-15), he went nine years before holding another press conference at which he took questions from the media—when he announced that John Fox's contract would not be renewed. One of the few times in which he has directly intervened in football matters came in the 2014-15 offseason, when he refused to resign Greg Hardy in the wake of domestic violence charges. Despite requests from players and coaches to let Hardy have another chance, Richardson said that he made the decision not to do so because "we do the right things."
It had long been presumed that Richardson intended to have his sons, Mark and the late Jon (who died in July 2013 from cancer), inherit the team. However, both stepped aside before the 2009 season. On January 16, 2013, WBTV in Charlotte reported that Richardson wants the team sold after he dies, but presumably only to someone who will keep the team in Charlotte.
Richardson was hospitalized in Charlotte at Carolinas Medical Center in early December 2008, one month after receiving a pacemaker. Richardson, who had a history of heart trouble and had undergone quadruple bypass surgery in 2002, was placed on a donor waiting list for a new heart two days later. He received a new heart on February 1, 2009, and has since recovered from the transplant.