Compton learned music from an early age as his great-grandfather was an old-time fiddler. Initially, Compton began playing the trombone but switched to guitar instead and later to mandolin playing old-time music with his cousin. He became interested in bluegrass music and eventually learned to play like Bill Monroe. At the Bean Blossom Bluegrass Festival in 1975, he finally met Monroe. After Compton had finished his education at the Meridian Junior College he moved to Nashville and joined Hubert Davis and the Season Travelers in 1977. Four years later, in 1981, he left Davis' band. He spent the early 1980s working as a cook, a printer, and only occasionally as a musician. In the mid-1980s, he joined the Nashville Bluegrass Band but left the band in 1988 due to a road accident where bass player Mark Hembree was injured. Compton moved to the Catskill Mountains in 1991 working as a cottage caretaker. The next year, he returned to Nashville to record an album with David Grier. Because session work was scarce, Compton began teaching mandolin. In 1995, he recorded with Bill Monroe's Bluegrass Boys. Compton joined John Hartford in the mid 1990s recording several albums together with him. In 2000, Compton returned to the Nashville Bluegrass Band as a replacement for the mandolin player Roland White.
In 2001 and 2002, Compton was nominated as IBMA Mandolinist of the Year. In 2002, O Brother, Where Art Thou?, an album containing one of his songs, was awarded the Grammy Award for best album. In recognition of his achievements, he received a commendation from the Mississippi State Senate.