Born in Styria, Weber started his professional career at Sturm Graz and joined Vienna giants Rapid Wien after the World Cup in 1978. He played a major part in the most successful of Rapid teams in the 1980-s, claiming the League crown four times, winning 4 domestic cups and most prominently losing the UEFA Cup Winners Cup Final in 1985 against Everton. He skippered Rapid in 1981 and from 1986 through 1989. He was voted in Rapid's Team of the Century in 1999.
At the end of his career he moved to SV Salzburg, with whom he won another league title during the club's most successful period. In 1994 he played with them in the UEFA Cup final against Inter Milan.