Thibodeaux played guitar from age six and fiddle from age twelve. He played in local dance halls at 13, and joined Julius Lamperez's Cajun band in 1949. He worked extensively as a studio musician for J. D. "Jay" Miller's Crowley Studio, working with The Clement Brothers among others. In 1952 he met Jimmy C. Newman, a frequent Grand Ole Opry performer; this began a long association between the two, including on Newman's first hit, "Cry Cry Darling". Alongside his time with Newman, Thibodeaux played in the studios behind Lefty Frizzell, George Jones, Jim Reeves, Slim Harpo, Carol Channing, Neil Young, and many others. His 1970 recording with Newman of the song "Lache Pas la Patate" was the first song sung in Cajun French ever to reach Gold status. In 1987, he released a solo album, The Cajun Country Fiddle of Rufus Thibodeaux.
In 1989, he contributed Cajun fiddle parts to the song, "Everything's Gonna Be Alright" from the Isadar album Dream Of The Dead. 
Rufus Thibodeaux is mentioned in a very popular French song of Michel Fugain : Les Acadiens.