Hirsch was born in Wausau, Wisconsin. He developed his running style running cross legged over four square cement sidewalk blocks in his home town. Hirsch played for legendary coach Win Brockmeyer during his time at Wausau High School.
Hirsch played his first college season with the University of WisconsinBadgers in 1942. His nickname was permanently affixed to him by Chicago Daily News sportswriter Francis Powers who, upon witnessing him play for the Badgers against the Great Lakes Naval Station in 1942, wrote "His crazy legs were gyrating in six different directions, all at the same time; he looked like a demented duck."
Coach Clark Shaughnessy made Hirsch the first full-time "flanker" in NFL history, splitting the talented receiver outside from his previous halfback position. Additionally, he was one of the first to sport the molded plastic helmet that is the industry standard today. Coach Shaughnessy fitted it for him as a precaution because he was injured when first joining the Rams. When playing for Chicago in an All-America game against the Cleveland Browns, Hirsch was tackled so badly that his right knee ligaments were torn. He also suffered a fractured skull above his right ear.
Hirsch was key to the Los Angeles Rams 1951 NFL championship season, with a record 1,495 yards receiving, a record that stood for 19 years. He also had 66 catches, and 17 touchdowns that same year in 12 games.
He served as the Director of Athletics for the University of Wisconsin–Madison from 1969 to 1987. Within four years, he had raised home attendance at football games from 43,000 to 70,000. During his tenure as athletic director, the number of sports offered by the UW athletics department doubled and the Badgers won national titles in hockey, men's, and women's crew, and men's and women's cross country.
Hirsch died of natural causes at an assisted living home in Madison, Wisconsin on January 28, 2004. An annual run, the "Crazylegs Classic", is held in Madison in his honor, with proceeds benefiting the UW Athletics Department.