Baumgartner made his major league debut on June 26, 1914 for the Phillies. He played in 15 games his rookie year, posting a 2–2 record and an earned run average of 3.28, along with two complete games and a shutout. The 1915 Philadelphia Phillies season saw Baumgartner as the main game finisher, pitching in 16 games and finishing a team-leading 12. He did not pitch in the 1915 World Series. Baumgartner only pitched in one game during the 1916 season, and did not pitch for the Phillies again until 1921. During the 1921 season, he pitched in 22 games, earning three wins against six losses. He pitched six games in 1922, he final season as a member of the Phillies.
In 1924, he joined the Philadelphia Athletics, and ended up having the best season of his career. During the 1924 Philadelphia Athletics season, he pitched in 36 games and started 16. He pitched 12 complete games, and posted an ERA of 2.88, which was fourth best in the American League. The following season, he pitched a career-high 17 games and earned an ERA of 3.57. After one more season with the Phillies as both a starting pitcher and relief pitcher, Baumgartner retired.