It was originally recorded by The Sparks Of Rhythm, a group Jones had been a member of when he wrote it, although he was not with them when they recorded it. In 1959, Jones recorded the song himself, in a version which had been reworked by Blackwell, who also produced the session. "Handy Man" went to number three on the R&B charts and number two on the Billboard Hot 100 in 1960, becoming a million seller. The song was a hit again in 1964 for Del Shannon, and again for James Taylor, whose slow version was a hit in 1977.
Allmusic critic Jason Elias compares Jones' original with Taylor's version stating that "where Jones’s version was chipper and a little obnoxious," Taylor's version "is so laid back it’s almost somnolent." Elias notes that Taylor's slowed down version has the benefit of allowing him to shade the words in new ways. Elias also praises Taylor's guitar playing.