Clyde McPhatter's roots were in the black church. The song is essentially the gospel song "Have Mercy, Jesus" sung in the call-and-response style of a gospel quartet, although it is in the straight twelve-bar blues form that gospel singers disdained. In the first chorus McPhatter simply follows the melody, but subsequently he freely improvises in the gospel style with short but spectacular melismas, stringing out phrases to overlap the backup singers responses, interjecting screams and yeahs, shouting a gospel funk. The backup band lays down the rhythm and provides the expected tenor sax solo.
The Dominoes' version of "Have Mercy Baby" was the definitive rhythm and gospel record. Other significant recordings of the song were made by The Bobbettes (1960) and by James Brown (1964), whose version charted #92 Pop.