After returning to Rapid in 2002, he decided to end his career with Major League Soccer franchise Los Angeles Galaxy in 2004. Under head coach, Sigi Schmid, Herzog played well but following a mid-season coaching change, he saw his playing time decrease and at the end of the season, Herzog announced his retirement from football on 10 November 2004.
A stylish attacking midfielder, Herzog was well known for his ability to score stunning free-kicks. He captained the Rapid Wien side and was chosen in Rapid's Team of the Century in 1999.
Herzog made his debut for Austria in an April 1988 friendly match against Greece and was a participant at the 1990 and 1998 World Cups, scoring one goal in the latter tournament, a spot kick against Italy. He earned 103 caps, scoring 26 goals, making him Austria's most capped player of all-time. Herzog surpassed striker Anton Polster in May 2002 when winning his 96th cap against Germany. His last international was an April 2003 friendly match against Scotland.