"That's Life" came to the attention of Frank Sinatra after it was recorded by blues singer O.C. Smith. The lyrics of "That's Life" were changed for Sinatra's version with the line "I've been a puppet, a pauper, a pirate, a poet, a pawn and a queen" changed to end with "a king" instead. Sinatra first performed the song on his television special A Man and His Music – Part II in 1966, where it was played in the fashion of a blues song with the backing music played on an electronic organ.
This arrangement was changed on the recorded version, with it instead conducted by Ernie Freeman and produced by Jimmy Bowen; the trio has previously worked together earlier in 1966 on "Strangers in the Night", which would go on to win Sinatra the Grammy Award for Best Male Vocal. For "That's Life", a fake gospel choir was added as a backing track and they assembled a number of backup musicians and singers for Sinatra's recording. After several attempts to record the track, they thought that they had it finished and Sinatra was pleased with it. However Bowen asked him to perform it again, which annoyed Sinatra – resulting in the performance that Bowen was looking for.
Bowen's vision for the rest of the album was to mirror "That's Life" onto the other songs so they all sounded similar, rather than fill it with what he viewed as typical Sinatra-style songs. This was as a result of his work on the Strangers in the Night album, where Bowen felt that the titular single didn't match the rest of the album, which was more of a classic Sinatra sound. So for the That's Life album, the other album tracks had similar brass accompaniments.
Both the album and the song proved major successes for Sinatra. The song was a number-four hit on the Billboard Hot 100 chart and reached number one on the Easy Listening chart for three weeks in December 1966/January 1967. Sinatra's cover of "That's Life" was later used in the 1993 film A Bronx Tale alongside his recording of Same Old Song and Dance. It was also featured in the 2004 video game Tony Hawk's Underground 2.