Ashman was too late to prevent that season's relegation but the following season saw a bounce back, and the season after that, the Third Division championship. Leading Division Two for much of the 1965–66 season, the club was disappointed to miss out on promotion by finishing third.
Ashman's exploits had not gone unnoticed and, in 1967, he joined West Bromwich Albion, leading them in a series of exciting cup runs, crowned by victory in the 1968 FA Cup. His new club was, however, impatient for further trophies and he was dismissed in 1971, hearing of the news from a waiter while on holiday in Greece.
Somewhat ironically, he went on to manage Olympiakos before returning to Carlisle in August 1972, leading the club to the First Division and a brief spell at the top of the entire English league.