He never got the chance to finish the 1973 season. Three days after a 14–0 defeat to the White Sox at Comiskey Park and Texas with a 47–91 record, he was dismissed on September 7. He was succeeded in the interim for one game by Del Wilber and in the longer term by Billy Martin, who had been fired by the Detroit Tigers on August 30. Short defended the change by telling reporters, "If my mother were managing the Rangers and I had the opportunity to hire Billy Martin, I'd fire my mother."
With his extensive background in player development, Herzog also was a general manager with both the Cardinals (1980–1982) and the California Angels. He succeeded Jack Krol as manager of the Redbirds in 1980, managed for 73 games, then moved into the club's front office as GM on August 26, turning the team over to Red Schoendienst. During the offseason, Herzog reclaimed the manager job, then held both the GM and field manager posts with St. Louis for almost two full seasons, during which he acquired or promoted many players who would star on the Cards' three World Series teams of the 1980s.
Herzog's style of play, based on the strategy of attrition, was nicknamed "Whiteyball" and concentrated on pitching, speed, and defense to win games rather than on home runs. Herzog's lineups generally consisted of one or more base-stealing threats at the top of the lineup, with a power threat such as George Brett or Jack Clark hitting third or fourth, protected by one or two hitters with lesser power, followed by more base stealers. This tactic kept payrolls low, while allowing Herzog to win consistently in stadiums with deep fences and artificial turf, both of which were characteristics of Royals Stadium (now Kauffman Stadium) and Busch Memorial Stadium during his managerial career.
Herzog also expressed an interest in becoming President of the National League when that job opened in 1986. The role eventually went to Yale University President A. Bartlett Giamatti, who also became the Commissioner of baseball in 1989. In a nationally-televised interview on NBC, after Giamatti accepted the job of NL President, Marv Albert jokingly asked Herzog if he would be interested in the job opening for President of Yale University. Herzog replied, "Well, you're trying to be funny now, Marv. I don't think that's funny at all."
After leaving the Cardinals in 1990, Herzog then held various front office and consulting posts with the Angels, including a brief stint (1993–1994) as general manager. Herzog and Jim Leyland were leading candidates to become manager of the Boston Red Sox following the 1996 season. Both rejected offers from the Red Sox, so the team hired Jimy Williams instead. Herzog was elected to the Baseball Hall of Fame by the Veterans' Committee on December 7, 2009, receiving 14 of a possible 16 votes. Herzog's induction into the Hall of Fame was on July 25, 2010. In addition, the Cardinals retired the number '24', which he wore during his managerial tenure with the club, in his honor on July 31, following his induction.Rick Ankiel was the last Cardinal to wear number 24.