Harnick began his career writing words and music to comic songs in musical revues. One of these, "The Merry Minuet", was popularized by the Kingston Trio. It is in the caustic style usually associated with Tom Lehrer and is sometimes incorrectly attributed to him.
Harnick began writing music while still in Carl Schurz High School. After his Army service, he graduated from the Northwestern University School of Music (1946-1949) with a Bachelor of Music Degree, and worked with various orchestras in the Chicago area. He then moved to New York City and wrote for many musicals and revues. He was friends with Charlotte Rae from college, and he went to see her one night at the Village Vanguard where she was singing a revue. Yip Harburg, who was one of Harnick's idols, heard she was singing a song of his and decided to come. He told Harnick that he enjoyed his writing, and to keep doing more. He gave him some advice, like work with a lot of composers and also if writing for Broadway, write character and comic songs, not ballads (Harnick followed both tips). Harburg told him to follow those steps because his old partner, Jay Gorney, had told them ballads was the key to success on Broadway.
Around 1956, Harnick met Jerry Bock, forming "what is arguably the most important musical partnership of the '60s."  Their first musical was The Body Beautiful, running for only 60 performances in 1958, but Fiorello! (1959) ran for 795 performances and won the Pulitzer Prize for Drama. Fiddler on the Roof (1964) "became one of the most cherished of all Broadway musicals."
Harnick wrote the libretto for the opera Coyote Tales, with music by Henry Mollicone, which received its world premiere at the Lyric Opera of Kansas City in March 1998. He wrote the book, music and lyrics to the musical Dragons, which was performed in 2003 at the Luna Stage (Montclair, New Jersey). He wrote the lyrics and co-wrote the book with Norton Juster for the musical The Phantom Tollbooth, based on the book by Juster. The musical premiered at the The Kennedy Center in 2007.
In honor of Harnick's vast influence on American music, on May 19, 1984 he was awarded the prestigious University of Pennsylvania Glee Club Award of Merit. Beginning in 1964, this award "established to bring a declaration of appreciation to an individual each year that has made a significant contribution to the world of music and helped to create a climate in which our talents may find valid expression."