He was involved in possibly the best era ever for Aberdeen who, under the management of Alex Ferguson, won the European Cup Winners' Cup in 1983 with a 2–1 win over Real Madrid in which the then 19-year old Black scored the first goal. Black's main asset was his balletic heading ability and scoring goals. He joined George Hamilton and Billy Williamson, becoming the third Aberdeen player to score three or more goals in a single game against Celtic. He made 180 league appearances (30 as substitute) and scored 70 goals for Aberdeen between 1981 and 1986.
That same year, Black joined Metz in France. After five seasons (1986–91) with Metz, during which he won the 1988 French Cup, and had two children, Black was forced to retire prematurely from the game due to a chronic back problem. In all, he made 85 appearances, scoring 26 goals.
Since retiring as a player Black has coached and managed, most notably as assistant manager at Celtic and manager at Coventry City and Motherwell. He joined Motherwell in October 2001. He was appointed Coventry manager in January 2004, following the resignation of Gary McAllister. Black was sacked from Coventry for "being inconsistent", despite a 5–2 win against Gillingham. This angered many fans who believed that Black had brought an attractive and highly effective style of football back to the club and was merely being moved-on for a bigger name (he was succeeded by Peter Reid, who proved to be a major disappointment). Black joined Birmingham City in July 2004 as assistant manager to Steve Bruce. He managed one game as caretaker manager, after Bruce joined Wigan Athletic, before leaving St Andrew's in November 2007.
Black was appointed caretaker manager of Sunderland on 30 November 2011, following the departure of Bruce. In his only game in charge, Sunderland were beaten 2–1 by Wolverhampton Wanderers. He left Sunderland on 6 December 2011, but quickly found a new job as he was named assistant manager to Steve Kean at Blackburn Rovers in January 2012.