The lyrics of "Southern Man" are vivid, describing the racism towards blacks in the American South. In the song, Young tells the story of a Southern white man (symbolically the entire white South) and how he mistreated his slaves. Young pleadingly asks when the South will make amends for the fortunes built through slavery when he sings, "I saw cotton and I saw black, tall white mansions and little shacks. Southern Man, when will you pay them back?" The song also mentions the practice of cross burning.
Merry Clayton's version of the song appeared on her self-titled 1971 album. Ironically, she later performed backing vocals on "Sweet Home Alabama", after some personal conflict.
Although the words "...southern man..." occur in "Sweet Home Alabama", the song of that name may not be the inspiration for the Lynyrd Skynrd song. In his book "Waging Heavy Peace: A Hippie Dream" Young stated that Lynyrd Skynyrd wrote "Sweet Home Alabama" not in response to "Southern Man", but rather to Young's song "Alabama". Young noted that Lynyrd Skynyrd's implied criticism was deserved because Young's lyrics to Alabama were condescending and accusatory, that he does not like how he used his words in the song, and the lyrics were not fully thought out and too easy to misconstrue.