After attending Fordham University in his native New York City, Murphy signed a professional contract with the New York Yankees. In 1934, his first full season with the Yankees, Murphy started 20 games (completing 10); for the remaining 11 years of his major league career, he would start only 20 games more, as he became one of the top bullpen specialists of his day. Moreover, his Yankees were one of the most powerful teams of all time, winning consecutive World Series championships from 1936 to 1939, and again in 1941 and 1943. Murphy's teammates included Lou Gehrig, Joe DiMaggio, Bill Dickey, Red Ruffing, Lefty Gomez—and, through 1934, Babe Ruth. Murphy spent his final year in the American League with the 1947 Boston Red Sox, the Yankees' arch-rivals.
Overall, he appeared in 415 games, winning 93, losing 53 (for a winning percentage of .637) with an earned run average of 3.50. He led the AL in wins for a relief pitcher seven times. While the save was not then an official statistic, Murphy four times led the AL in that category. In eight World Series games and 16⅓ innings (spread over six different Series), Murphy won two games, lost none, saved four, and posted an ERA of 1.10. Johnny Murphy was on 7 World Series winning teams, the most of any pitcher in history.
When his playing days ended, Murphy stepped immediately into the Boston front office when owner Tom Yawkey appointed him Director of Minor League Operations. Murphy spent 13 seasons running the Red Sox' farm and scouting systems until his dismissal following the 1960 season. In 1961, he joined former Yankees farm director and general manager George Weiss in the front office of Gotham's National League expansion team, the New York Mets.