He was involved in possibly the best era ever for Aberdeen who, under the management of Alex Ferguson, won the European Cup Winners' Cup in 1983 with a 2–1 win over Real Madrid in which the then 19-year-old Black scored the first goal. Black's main asset was his balletic heading ability and scoring goals. He joined George Hamilton and Billy Williamson, becoming the third Aberdeen player to score three or more goals in a single game against Celtic. He made 180 league appearances (30 as substitute) and scored 70 goals for Aberdeen between 1981 and 1986.
That same year, Black joined Metz in France. After five seasons (1986–91) with Metz, during which he won the 1988 French Cup, and had two children, Black was forced to retire prematurely from the game due to a chronic back problem. In all, he made 85 appearances, scoring 26 goals.
Since retiring as a player Black has coached and managed teams. He was a coach and assistant manager at Celtic. Black was appointed manager of Motherwell in October 2001. He was appointed Coventry City manager in January 2004, following the resignation of Gary McAllister. Black was sacked by Coventry for "being inconsistent", despite a 5–2 win against Gillingham. This angered many fans who believed that Black had brought an attractive and highly effective style of football back to the club and was merely being moved-on for a bigger name (he was succeeded by Peter Reid, who proved to be a major disappointment).
Black joined Birmingham City in July 2004 as assistant manager to Steve Bruce. He managed one game as caretaker manager, after Bruce joined Wigan Athletic, before leaving St Andrew's in November 2007. He was assistant manager to Bruce at Sunderland from June 2009 and was appointed caretaker manager following the departure of Bruce. In his only game in charge, Sunderland were beaten 2–1 by Wolverhampton Wanderers. Black left Sunderland on 6 December 2011.