Although never a regular starter for Wolves, he made five appearances in 1953–54, the season of the club's first championship. Lacking playing time for Wolves, he moved to Midlands neighbours Aston Villa in November 1953.
Baxter made over 100 appearances for Villa before retiring in Summer 1957. He stayed with the club to serve them as coach and assistant manager until 1967. He then moved back to his native Scotland where he had short spells in charge of both East Fife and Raith Rovers.