Swain began his professional career with Londoners Chelsea, signing from Wycombe Wanderers in the summer of 1973. As he had been a non-contract player with Wycombe, having joined them from college football, his signing-on fee was just £500. He made his League debut for Chelsea in 1974. His most successful season with the club came in 1976-77, when his strike partnership with Steve Finnieston helped Eddie McCreadie's young side win promotion back to the First Division. He remained with Chelsea until 1979, when they were relegated, and then signed for Aston Villa for £100,000.