In October 1961, Gemmell joined Celtic from Coltness United. He was one of the 'Lisbon Lions' who won the 1967 European Cup Final, in which Gemmill scored a spectacular equalising goal. Ironically, Gemmell should not have been in position to score the goal, as he had ignored team orders for one full-back to stay in defence at all times. He also scored in the 1970 European Cup Final against Feyenoord and is currently one of only two British footballers to score in two different European Cup Finals, the other being Phil Neal of Liverpool. He made 418 appearances for Celtic and scored 64 goals. This total comprised 247 league (37 goals), 43 cup (5 goals), 74 league cup (10 goals) and 54 European (12 goals) appearances. His record for penalties was 34 goals from 37 attempts.
Gemmell later managed Dundee and Albion Rovers. He signed Jimmy Johnstone, his former teammate at Celtic, for Dundee. Johnstone later admitted that he took liberties during this time because Gemmell was his friend.