Thomas grew up in Russellville, Alabama. His father was the town's superintendent of education, his mother was a music teacher. Thomas credits them, along with his brother, two sisters and his extended family, for his strong sense of character.
He was educated at Harvard University where he earned three degrees in biological sciences with a concentration in botany.
He served as a professor of biology at the University of Alabama from 1966 until his appointment as vice president for student affairs in 1969.
Thomas took the position of chancellor at North Carolina State University in 1975; the school's ninth chief executive. While in office, enrollment at the university surpassed 20,000 for the first time. He also oversaw the establishment of the School of Veterinary Medicine. Thomas resigned as chancellor in 1981. A partial manuscript collection related to Joab Langston Thomas is housed in the NCSU Libraries Special Collections Research Center in D.H. Hill Library.
In 1981 Thomas returned to the University of Alabama to serve as the school's president, an office he held until 1988. The most notable hire of his presidency at Alabama was that of Ray Perkins to succeed Bear Bryant as the school's football coach in December 1982. Bryant died of a heart attack four weeks after coaching his final game on January 26, 1983. Thomas later hired Bill Curry to succeed Perkins in 1987 when Perkins left to become coach of the National Football League's Tampa Bay Buccaneers.
He also served as president of Pennsylvania State University from 1990-1995. The Thomas Building on Penn State's University Park campus is named in his honor, and houses parts of the Eberly College of Science, including the Department of Statistics.