Born in Mannheim, Schaaf arrived at SV Werder Bremen's youth academy in 1972, turning professional six years later. After a slow start with the first team, where he made only 21 league appearances in four years combined – 19 of them coming in 1980–81 in the second division – he eventually became an important squad member; he made his debut in the Bundesliga on 18 April 1979, in a 0–3 away loss against VfL Bochum.
On 14 December 2009, Schaaf signed a new contract with Werder Bremen. He led the side to the third place in the league and the playoff stages in the 2010–11 Champions League, as well as to a second straight German Cup final, which was lost to Bayern Munich.
Schaaf left Werder on 15 May 2013 by mutual consent after finishing a disappointing fourteenth in the domestic championship, ending 14 years in charge of the club and ending his 41-year association with the club since joining as an 11-year-old youth player. He oversaw 645 games as a coach during his stint, finishing with a record of 308 wins, 138 draws, and 199 losses and leading it to six major trophies and six appearances in the Champions League, and was linked to the organization for four decades since his days as a youth player. During the press conference where he announced his resignation, he spoke of his admiration of the club and the joy of his time spent at the Weserstadion, saying, "I had an extraordinary time here, connected with a lot of positive experiences and great successes. I would like to thank everyone who accompanied me along the way and supported me. I wish Werder Bremen a successful future."
On 21 May 2014, after one year out of football, Schaaf was appointed head coach of Eintracht Frankfurt, signing a two-year contract. During the course of his first season he led his team to a ninth-place finish, being in charge of his 500th Bundesliga match in the process.
Schaaf resigned on 26 May 2015, with a record of 12 wins, 10 draws and 14 losses from 36 games.
Schaaf was appointed as the head coach of Hannover 96 on 28 December 2015, signing an 18–month contract and being formally introduced to the media after his first training session on 4 January 2016. He took over a team that was in 17th place, after they took 14 points from a possible 51 when he was hired.