Mackay was born in Edinburgh, and began his playing career with the club he supported as a boy, Heart of Midlothian. He won all three Scottish domestic honours with the club. He captained the side in 1957–58, when the club won the Scottish league title. The team broke a British league goalscoring record, with 132 goals for and only 29 conceded.
In 1971 he was appointed player-manager of Swindon Town but left after just one season to take charge of Nottingham Forest. He remained at the City Ground until October 1973, when he returned to Derby as manager following Clough's resignation. In his first season Derby finished third in the table. In his second season in charge of Derby, he guided the team to the 1974–75 league title. The following season, he managed the club to a respectable fourth-place finish in the league, the semifinals of the FA Cup, and an unfortunate extra time second-round exit to Real Madrid in the 1975–76 European Cup. At one stage the side had been in the running for the Double. Mackay was sacked in November 1976 after a poor start to the 1976–77 season. A newspaper headline reading "Mackay Gets The Boot" was used as a visual prop in the British television situation comedyPorridge, which featured a prison officer named Mackay.
He then had an uneventful spell as Walsall manager from March 1977 to August 1978. This was followed by nine years coaching in Kuwait. He returned to the UK and was appointed manager of Doncaster Rovers in 1987, a year after being linked with the Scotland manager's job (which ultimately went to Andy Roxburgh). Mackay's reign at Belle Vue lasted two seasons before he moved to Birmingham City, who had just been relegated to the third tier of the league for the first time in their history. His task was simple – to get Birmingham promoted to the Second Division. But he was unsuccessful in trying to achieve this and resigned in 1991. After that, he returned to the Middle East for two years managing Zamalek, a Cairo club team, with which he won the Championship, and then a further three years in Qatar before retiring from football altogether in 1997.
In 2004 The Real Mackay was published, an autobiography written with Martin Knight. Mackay had previously published Soccer My Spur in the early 1960s.
Mackay was made an inaugural inductee of the English Football Hall of Fame in 2002 in recognition of his impact on the English game as both a player and manager, and in 2006 became an inaugural inductee of the Heart of Midlothian Hall of Fame in recognition of his success as a player in the 1950s.
George Best (1946–2005), of Manchester United, one of Tottenham's fiercest rivals in the 1960s, described Mackay as "the hardest man I have ever played against – and certainly the bravest".