In 1951, he signed for Carlisle United for whom he made 207 appearances, scoring 98 goals. In 1958, suffering from cartilage damage, he retired from playing to manage the poultry farm of one of the club's directors. As a recreation, he managed amateur team Penrith and when Third Division Carlisle found themselves in need of a manager in February 1963, the club offered Ashman the job. During his playing career he played cricket for Cumberland in the 1956 Minor Counties Championship.
Ashman was too late to prevent that season's relegation but the following season saw a bounce back, and the season after that, the Third Division championship. Leading Division Two for much of the 1965–66 season, the club was disappointed to miss out on promotion by finishing third.
Ashman's exploits had not gone unnoticed and, in 1967, he joined West Bromwich Albion, leading them in a series of exciting cup runs, crowned by victory in the 1968 FA Cup. His new club was, however, impatient for further trophies and he was dismissed in 1971, hearing of the news from a waiter while on holiday in Greece.
Somewhat ironically, he went on to manage Olympiakos before returning to Carlisle in August 1972, leading the club to the First Division and a brief spell at the top of the entire English league.