Ashman played 662 games for Norwich City, scoring 56 goals. 590 of those games were league appearances – a club record. He made his debut at Carrow Road against Aldershot on 4 October 1947. Ashman was a centre-forward at that early stage of his career, but went on to play at full-back for many years. He was the captain of the Norwich team that reached the semi-finals of the FA Cup in 1959 as a third division side, won promotion to the second division in 1960 and won the Football League Cup for the first time in 1962.
When George Swindin resigned as manager in November 1962, Ashman was appointed acting manager and was eventually given the job on a permanent basis on Boxing Day 1963. By then, he had played his last game for the club on 19 October 1963. Ashman remained manager until the end of the 1965–66 season, when his departure ended twenty years of service to the club.
Ashman was manager of Scunthorpe United for six years from 1967 to 1973 and a further five years from 1976 to 1981. In between those two spells, he was Grimsby Town manager for two years.