Brown earned seven letters at North High School in Denver, where he starred in football, basketball and baseball. He played basketball and baseball at the University of Northern Colorado. After college, he coached football and baseball at Arvada High School in Arvada, Colorado, and he also coached baseball at the University of Colorado Boulder and Metro State College. From 1969 to 1977, Brown had a career as a basketball referee, officiating six Final Fours. Brown served as commissioner of the Colorado Athletic Conference in the early 1990s. In 2007 he was named the new commissioner of the Mountain Collegiate Baseball League (MCLB).
Brown began his broadcasting career in the late 1970s on KHOW radio. For the past 25 years he has been on the air with Joe Williams, currently hosting the show The Fan. Brown and Williams later co-authored The Great Book of Denver Sports Lists. Brown retired from hosting "The Irv and Joe Show" on Mile High Sports Radio AM 1340 with longtime partner Joe Williams, on April 7, 2016, ending a 44-year media career. He was also an analyst for the NBA on ESPN during the 1980s.
Brown was inducted into the Colorado Sports Hall of Fame in 1997.
- Henderson, John (April 7, 2013). "Irv Brown remembers his longtime career as college basketball referee". The Denver Post. Denver, Colorado. Retrieved January 11, 2016.