McCarthy's career winning percentages in both the regular season (.614) and postseason (.698, all in the World Series) are the highest in major league history. His 2,126 career victories rank eighth all-time in major league history for managerial wins, and he ranks first all-time for the Yankees with 1,460 wins.
After a brief managing stint in 1913 while playing in Wilkes-Barre, he resumed his managing career with Louisville in 1919, leading the team to American Association pennants in 1921 and 1925 before being hired to manage the Chicago Cubs for the 1926 season. He turned the club around, guiding them to the 1929 NL title, but was fired near the end of the 1930 season. He was then hired to manage the 1931 season for the New York Yankees.
With the Yankees, his strict but fair managing style helped to solidify the team's place as the dominant franchise in baseball, with a World Series title in 1932, and four consecutively from 1936 to 1939; the Yankees became only the third team – and the first in the AL – to win four straight pennants, and the first to win more than two World Series in a row. McCarthy struggled to control his emotions at the moving testimonial held for Lou Gehrig at Yankee Stadium on July 4, 1939. After describing Gehrig as "the finest example of a ballplayer, sportsman, and citizen that baseball has ever known", McCarthy could stand it no longer. Turning tearfully to Gehrig, he said, "Lou, what else can I say except that it was a sad day in the life of everybody who knew you when you [...] told me you were quitting as a ballplayer because you felt yourself a hindrance to the team. My God, man, you were never that."
Despite the loss of Gehrig, the Yankees went on to win three more AL crowns from 1941 to 1943 before McCarthy resigned in May 1946, partially due to conflict with new club operator Larry MacPhail. McCarthy returned as manager of the Boston Red Sox from 1948 to June 1950, but was unable to capture a pennant despite reaching a one-game playoff with the Cleveland Indians in 1948.
Despite his teams' great performance, McCarthy was not without his detractors, who believed he was simply fortunate enough to be provided with great talent and was not a strong game tactician. During his peak period from 1936 to 1943, when the Yankees won seven pennants in eight seasons, White Sox manager Jimmy Dykes described McCarthy as a "push-button" manager. Yet McCarthy was an outstanding teacher and developer of talent, and was particularly adept at handling temperamental players such as Babe Ruth, who had hoped to become New York's manager and resented a team "outsider" being hired.
While managing, McCarthy utilized a low-key approach, never going to the mound to remove a pitcher or arguing with an umpire except on a point of the rules, preferring to stay at his seat in the center of the dugout. He also declined to wear a numbered uniform with the Yankees and Red Sox.[not in citation given]
In order to draw attention to his presumed masterful leadership of the Yankees, McCarthy was given the nickname of "Marse Joe" by sportswriters. "Marse" is a Southern English rendition of the word "master". McCarthy's success throughout his career was such that in 32 years of managing, his 1922 Louisville club was the only team which finished either with a losing record or below fourth place.
McCarthy was named Major League Manager of the Year by The Sporting News in 1936 – the first year the award was given – and again in 1938 and 1943.
In a 1969 poll by the Baseball Writers Association of America to commemorate the sport's professional centennial, McCarthy finished third in voting for the greatest manager in history, behind John McGraw and Casey Stengel. In a similar BBWAA poll in 1997 to select an All-Century team, he finished second behind Stengel. On April 29, 1976, the Yankees dedicated a plaque for their Monument Park to McCarthy. The plaque calls him "One of baseball's most beloved and respected leaders." In honor of his commitment to Buffalo, McCarthy became a charter member of the Buffalo Baseball Hall of Fame in 1985.