Joe was born in Quincy, Massachusetts on March 27, 1929 and grew up in Somerville, Massachusetts. He received his bachelor's degree in French from Boston College in 1951 and his master's degree in French literature, also from Boston College, in 1953. He planned to be a professor of French literature but began doing freelance reviews and found that he had a talent for journalism.
He once told Washingtonian magazine: "To be the primary critic of a monopoly newspaper is an overwhelming role. You have to tread softly and be fully aware that your taste is not the only valid taste. All these years, I pasted in the front of my mind that there are many ways to be good." 
In addition to serving as chief music critic for the Post, McLellan wrote a chess column, wrote for Book World and covered White House parties and other society events for the Style section. He covered world chess matches, and edited a syndicated column by the Czechoslovakian American grandmaster Lubomir Kavalek.
He reputedly taught himself to write as a youngster by reading How to Write, a book that came as a bonus with a set of encyclopedias he bought with money earned from his paper route.
McLellan died of kidney failure on December 26, 2005. He had continued to review concerts for The Washington Post until the fall of 2005.