Part of the guitar riff uses a variant on the clave rhythm.
In 2010, examination of the master recording tape for the track by recording engineer Chris Baseford, revealed that, similar to most bands of the time, the band played together while tracking in the studio instead of overdubbing the instrumental elements; some guitar amp sound could be heard leaking into the drum tracks and some drum tracks leaking into the guitar track. The production on the song was described as "pretty standard". Aside from the drums, panned slightly off center, there were some additional percussion, tambourine, and handclap overdubs. Baseford described the bass performance and sound on this song as "top notch" with Tiran Porter playing the melodic line using a pick and plugging directly into the mixing board.
Like many songs by Johnston, the music was composed and developed before the lyrics were written. It started with a guitar riff that he and drummer John Hartman developed into a jam with a chord structure. Johnston said that the lyrics were influenced by a piano part in the performance. According to Johnston, "...I really owe Billy Payne for the words because he played this wacky bridge that started the thinking process with this wacky sheriff, samurai swords, and all that."
The song is based on a real town in Texas. Johnston claims that he thought he had created a fictional town called "China Grove" near San Antonio, Texas, and later learned it really exists. He explained that the band had been on tour passing through China Grove on the way to or from San Antonio, and he had seen a road sign with the name, but forgot about it.