Born in Bolton, Lancashire, Mangnall played amateur football as a goalkeeper and was a director at Bolton Wanderers. He started his career in football management with Burnley in March 1900, when he was hired as the club's second official manager following the departure of Harry Bradshaw almost a year earlier. The side were struggling when he joined the club, and with just one month of the 1899–1900 season remaining, relegation to the Second Division seemed almost inevitable. Their relegation was confirmed on 28 April 1900 after a 0–4 defeat away at Nottingham Forest. He signed three players, Henry Collins, Jimmy Lindsay and George Lockhart, in the close season. The team achieved a third-placed finish in their first season in the Second Division and finished ninth the following year. The team's fortunes suffered a swift decline in the 1902–03 season, and Burnley finished bottom of the Football League and were forced to apply for re-election. Burnley were re-elected and Mangnall started the next campaign as manager, but he left the club in October 1903 to join Second Division rivals Manchester United.
He was hired as the third secretary of Manchester United and the second after the club changed its name from Newton Heath (the term 'manager' was not used at United until the arrival of Jack Robson). He was to become the first successful manager at United and remains the third most successful manager at the club. He started his management at Manchester United in 1903 and saw his team narrowly miss promotion in his first two seasons before success at the third attempt. In the first season in the First Division the club finished in mid-table. However, in only their second season in the First Division, he managed United to their first ever League Championship in 1907–08 by a nine-point margin over Aston Villa. The following year saw United drop below mid-table but the club won its first FA Cup with a 1–0 victory in the final against Bristol City, the winner scored by Sandy Turnbull. The following season saw no new silverware but the team improved its league position to fifth. The following season the club were champions again, again beating Aston Villa into second place, this time by just one point; that would be club's last league championship for over 40 years – its longest ever run without a league title. The next season was his last in charge at the club; he would eventually leave for a similar position at Manchester City.
Mangnall moved to Manchester City directly from United. His last game in charge of the Reds was the Manchester derby of September 1912 against City when it was already known he would become City manager. The Blues won 1–0 at Old Trafford and the media focused on Mangnall's delight.
He managed City from 1912 until 1924, the interruption of World War I would mean his spell there only covered eight Football League seasons however the Club did find success in wartime regional tournaments. His best season with the club was 1920–21, when City would finish runners up in the First Division, the top level of English football at the time.
Mangnall's place in Mancunian football is significant as many believe he was the instigator behind United's move to Old Trafford and City's move to Maine Road.