It was composed by Mann and Hilliard during a post-midnight session at Hilliard's New Jersey home. Mann was about to depart for New York when Hilliard insisted he remain to try some impromptu songwriting. Mann reluctantly agreed and eventually came up with the tune, to which Hilliard quickly wrote a lyric.
Barbra Streisand has covered the song twice. On her 1991 retrospective, Just For the Record, Barbra sang a medley of "When You Gotta Go" and "Wee Small Hours of the Morning" (from a 1969 live performance). In 2009, Streisand recorded a new, studio version of "Wee Small Hours" for her album, Love is the Answer, produced by jazz artist Diana Krall.