In October 1961, Gemmell joined Celtic from Coltness United. He was one of the 'Lisbon Lions' who won the 1967 European Cup Final, in which Gemmill scored a spectacular equalising goal. Ironically, Gemmell should not have been in position to score the goal, as he had ignored team orders for one full-back to stay in defence at all times. He also scored in the 1970 European Cup Final against Feyenoord and is currently one of only two British footballers to score in two different European Cup Finals, the other being Phil Neal of Liverpool. He made 418 appearances for Celtic and scored 64 goals. This total comprised 247 league (37 goals), 43 cup (5 goals), 74 league cup (10 goals) and 54 European (12 goals) appearances. His record for penalties was 34 goals from 37 attempts. In his book, Lion Heart, Gemmell revealed that during his time at Celtic he was on the receiving end of sectarian abuse from fellow Celtic players. Gemmell stated that he and team-mate Ian Young had been the target of colleagues who had wanted an all-Catholic team.
Gemmell later managed Dundee and Albion Rovers. He signed Jimmy Johnstone, his former teammate at Celtic, for Dundee. Johnstone later admitted that he took liberties during this time because Gemmell was his friend.