The song is about a man who is torn between two women that he loves: one being his significant other and the other being his mistress. This is apparent when Doctor Zhivago is referenced in the second verse of the song. In the end, the man chooses to say goodbye to his mistress because it is only fair to his significant other, who has always trusted him. This becomes clear in the first verse of the song when he says, "I've got somewhere else to be, promises to keep/Someone else who loves me and trusts me fast asleep." However, as he is saying goodbye to the mistress, he has to hide his true feelings (love) from her; hence, the title of the song: "The Hardest Thing". As he says goodbye to his mistress, he thinks, "It's the hardest thing I'll ever have to do to look you in the eye and tell you I don't love you." However, he knows that their love is real and that they will meet again when the time is right: "I know that we'll meet again/Fate has a place and time/So you can get on with your life."
The music video takes place inside a boxing arena. Nick Lachey, the protagonist in the video, is the man who is torn between two women. Nick is a boxer in the video, and his mistress is a show girl. His significant other is not in the video.