A former centre back who played several seasons for his beloved team, A.S. Roma, other than Spal and Ascoli, where he played for nine season, Mazzone left the playing career in season 1968–1969, to become the manager of Ascoli itself, in Serie C, helping the club to win the Serie C title in 1972.
Already popular because of his playing history for the team, Mazzone gained even more popularity by leading the team for twelve years, up to the first Serie A appearance in the team history. Successively, Mazzone coached several Serie A and Serie B teams, such as Fiorentina (achieving his personal best result in Serie A, a third-place finish during the 1976-77 Serie A season, also winning the Anglo-Italian League Cup in 1975), Catanzaro, Bologna F.C. 1909, Lecce, Pescara, and Cagliari (1991-93), leading the team to its first appearance in a European competition since Gigi Riva's times.
After his successes at Cagliari, Mazzone fulfilled his dream, being called to coach AS Roma for the 1993-94 Serie A season. He remained at the helm of AS Roma for three seasons, however without being able to obtain any notable triumph.
He then went back to coach Cagliari in 1996, then coached Napoli (1997-98), Bologna again in 1998 (winning the 1998 UEFA Intertoto Cup, qualifying for the 1999-2000 UEFA Cup, and reaching the semi-final of the 1998-99 UEFA Cup), Perugia (1999-2000), and subsequently Brescia in 2000, where he had the opportunity to have Roberto Baggio among his players, During his first season with the club, he helped the club to avoid relegation, leading them to their best ever Serie A finish to qualify for the 2001 UEFA Intertoto Cup, where they reached the final. Mazzone was at the centre of controversy when, on 30 September 2001, during a league match against team rivals Atalanta, after his team tied the score in the final minutes, he ran for about 70 meters and screamed towards the Atalanta supporters, guilty of having offended him for all the match long with personal insults. That season Brescia reached the semi-finals of the Coppa Italia, their best ever result in the competition. In 2002, Mazzone was awarded the Panchina d'oro award, in honour of his career.
Mazzone left Brescia in 2003 in order to coach Bologna for the third time on his long career. However, his adventure lasted only two season, as the team was surprisingly relegated in Serie B at the end of the 2004-05 season. It was the first relegation ever in Mazzone's career, and it convinced him to take a break for a while.
However, on 7 February 2006, at the age of 68 years and 11 months, Mazzone accepted an offer of Livorno for filling the coaching office left vacant by Roberto Donadoni, who resigned the day before. He left the job at the end of the season.
Mazzone is the most experienced coach in the Italian football panorama, having coached on more than 1,000 professional matches. On 18 March 2006, Mazzone has become the Italian manager who coached the highest number of Serie A matches ever, equalling and then overcoming the past record held by Nereo Rocco.
Roberto Baggio often defined several times Mazzone as one of the best coaches he met in his playing career. Baggio himself, in a radio interview on 18 February 2006, declared that Mazzone asked him to join him in a playing football comeback in Livorno; however, Baggio claimed that, even if he would have given him an affirmative answer because of his gratitude for what Mazzone did for him, he was forced to refuse because of his physical troubles.
Pep Guardiola said that he learnt a lot from Mazzone during his Brescia years.
He is often nicknamed Sor Carletto because of his Roman origins, and the strong accent typical of the inhabitants of the Italian capital city.
Despite having coached around the whole country for 35 years and more, Mazzone has always held his residence in Ascoli Piceno, home of Ascoli Calcio, the team in which he started his long managing career.