In his younger days, Ellis played as a centre-half for Sheffield Wednesday in the 1960s. As a teenager he made his debut in the 1966 FA Cup final against Everton, in which his team lost 3–2.. He then moved on to play for Mansfield Town, Lincoln City, and Watford. It was at Vicarage Road that he became coach and assistant manager to future England boss Graham Taylor. With Ellis' help, Watford climbed the divisions before turning down a new contract in favour of trying his hand at management. When the Blackpool job was advertised, Ellis applied and was given the job on 1 June 1982.
Ellis was manager at the seaside for seven years, from 1982 to 1989, with no prior managerial experience, but the new board at Bloomfield Road were confident in his ability. Despite finishing in the bottom four of the Fourth Division in his first season, thus forcing the club to apply for re-election to the Football League, Ellis managed to turn the side around and gain promotion two years later with only limited funds handed to him. Although he kept the Tangerines in the Third Division, they were unable to make further progress and Ellis' contract was terminated on 28 March 1989, with relegation looking a possibility. At the time of his departure, Ellis was the third longest serving Blackpool manager in terms of Football League games in charge.