Brown earned seven letters at North High School, where he starred in football, basketball and baseball. He played basketball and baseball at the University of Northern Colorado. After college, he coached football and baseball at Arvada High School, and he also coached baseball at the University of Colorado and Metro State College. From 1969 to 1977, Brown had a successful career as a basketball referee, officiating six Final Fours. Brown served as commissioner of the Colorado Athletic Conference in the early 1990s.
Brown began his broadcasting career in the late 1970s on KHOW radio. For the past 25 years he has been on the air with Joe Williams, currently hosting the show The Fan. Brown and Williams later co-authored The Great Book of Denver Sports Lists.
Brown continues to host "The Irv and Joe Show" on Mile High Sports Radio AM 1340 with longtime partner Joe Williams. He was also an analyst for the NBA on ESPN during the 1980s.