Shay was born in Chicago, Illinois. While still young, he studied the piano at the Bush Conservatory of Music in Chicago. He eventually moved to New York City to become a songwriter. His first composition was "Do You, Don't You, Will You, Won't You," published in 1923. In 1925 he joined ASCAP (American Society of Composers, Authors and Publishers), and remained a member for 63 years. In 1929 he co-authored his most famous song, "When You're Smiling" (As with many other of his songs, this was a collaboration with Joe Goodwin and Mark Fisher; see Shay, Fisher, and Goodwin).
In the 1930s, Metro-Goldwyn-Mayer hired him to become their music director, and Shay and his wife Doris moved from New York to Hollywood. In that capacity, he hired Bing Crosby, who was paid $50 a day by MGM for his first picture.