Brown's NFL career began promisingly, as he played a part in the Steelers' championship team of 1975. The following spring, Brown was chosen by the Seahawks in their expansion draft. He spent ten years with the team, setting numerous franchise records. A cornerback, Brown tied an NFL record with two interception returns for touchdowns in a single game (4 November 1984 vs. the Kansas City Chiefs). He was selected to the Pro Bowl following the 1984 season.
After leaving the Seahawks in 1986, he played three more years with the Packers, finally retiring in 1990. His sixty two career interceptions ranked seventh all-time at his retirement, and his fifty interceptions for Seattle remains the team record. He was elected to Seattle's elite "Ring of Honor" in 1992, and that same year he returned to the Seahawks as their defensive backs coach, a position he held until 1998. In 2000, he was named to his alma mater Michigan's All-Century team. In 2001 Brown was hired as the defensive backs coach for Texas Tech University in Lubbock. During his tenure the Red Raiders would go on to beat Clemson 55-15 in the 2002 Tangerine Bowl, beat Navy 38-14 in the 2003 Houston Bowl, beat California in the 45-31 in the 2004 Holiday Bowl and lose to Alabama 13-10 in the 2006 Cotton Bowl Classic.