Swofford was born on December 6, 1948 in North Wilkesboro, North Carolina, in the foothills of the Blue Ridge Mountains. While at Wilkes Central High School, he played as quarterback for the Wilkes Central Eagles football team and was twice selected to the all-state football team. He was awarded a prestigious Morehead Scholarship to attend the University of North Carolina at Chapel Hill, where he also played for the football team. He holds a master's degree in Sports Management from Ohio University. From 1980 to 1997 he was the athletic director for the University of North Carolina at Chapel Hill.
In 1997 Swofford was named the Commissioner of the ACC. During his tenure he has doubled the ACC's annual revenue, served as Chairman of the Bowl Championship Series in college football in 2000 and 2001 and expanded the ACC to include Boston College, Virginia Tech, and the University of Miami, and by 2013, to include Syracuse, Pitt, and University of Notre Dame, and also University of Louisville by 2014.
Swofford is the brother of William Oliver Swofford (1945–2000), a pop singer professionally known as Oliver who performed from the late 1960s through the late 1970s. Swofford and his family live in Greensboro, North Carolina, where the ACC has its headquarters.