The club was founded 1907 and played in the second division of the Westkreis-Liga just before the First World War. Waldhof became part of the Kreisliga Odenwald in 1919 and won this league in 1920 and 1921. In each of those two seasons, the club failed to advance in the Southern German championship because it was grouped with all-powerful 1. FC Nuremberg at the time. The club took out a Bezirksliga Rhein championship in 1924 before joining the Bezirksliga Rhein-Saar in 1927, where it won five out of the next six division titles without ever performing particularly well in the Southern championship.
Its enjoyed its best performances in the Gauliga Baden, one of sixteen top-flight divisions established through the 1933 re-organization of German football under the Third Reich. Waldhof dominated the division through the 30s and into the early 40s by capturing the title five times and consistently finishing well up the table. They were unable, however, to translate that into success at the national level. Their best result came in 1940 when they went out in a semi-final match against FC Schalke 04, the dominant side of the era, before settling for fourth place after losing a consolation round match to Rapid Vienna.
After World War II, Waldhof picked up play in the Oberliga Süd where they earned mid-table results until being relegated to the 2nd Oberliga Süd in 1954. They bounced up and down between first and second division play until the formation of the Bundesliga, Germany's new professional football league, in 1963. The next season saw them in the tier II Regionalliga Süd alongside local rivals VfR Mannheim. A string of unimpressive results finally led to relegation in 1970 to the Amateurliga Nordbaden (III).
SV Chio Waldhof Mannheim ca. 1972–78.
Support from a new sponsor, snack chip maker Chio, revived the team and helped their return to the second division where they played as SV Chio Waldhof Mannheim from 1972 to 1978. They continued to play as a middling side there until an unexpected breakthrough to the Bundesliga in 1983. Waldhof spent seven seasons competing at the top flight until a 17th place finish saw the club relegated at the end of the 1989–90 season. They delivered another seven seasons as a decent 2.Bundeliga club until slipping to the Regionalliga Süd for two seasons in 1997–99. A merger with VfR Mannheim was considered in 1998 but the club walked away from a deal at the last minute. Their return to the 2.Bundesliga in 1999 after a season long struggle with Kickers Offenbach was cut short in 2003 when financial irregularities saw the German Football Association deny the team a license, dropping them to the Oberliga Baden-Württemberg (IV). Another attempt at a merger with VfR failed that same year. The club played in the Oberliga Baden-Württemberg until 2007–08, when a third place finish allowed them qualification for the Regionalliga.
After coming fourth in the Regionalliga Süd in 2008–09, the club was moved across to the Regionalliga West in 2009–10 in an effort to balance out the three Regionalligas.
After just two seasons in the Regionalliga, Waldhof again had their license withdrawn in 2010 and were demoted back to the Oberliga Baden-Württemberg, now the fifth level of German football, despite having finished clear of the relegation zone on the pitch with the league's smallest budget. Waldhof Mannheim spend only one year in the Oberliga Baden-Württemberg, winning this league in 2010-11 and advancing directly back to the Regionalliga. On 11 June 2011 they defeated FV Illertissen 6-0 in their final season match to clinch promotion and also set a new fifth-division attendance record, with 18,312 spectators seeing the match. It surpassed the previous record, the 2009 Leipzig derby, by more than 3,000 spectators.
At the end of the 2011-12 season the club was grouped into the new Regionalliga Südwest, which replaced the Regionalliga Süd in the region.
The SV Waldhof II, historically also referred to as SV Waldhof Amateure, rose to the tier-IV league Verbandsliga Nordbaden in 1986 and remained there until gaining promotion to the Oberliga Baden-Württemberg in 2001. After two seasons in the Oberliga with good results, the team had to be withdrawn due to the forced relegation of the first team. In the 2007–08 season, the team narrowly missed out on Verbandsliga promotion when it finished second on equal points to the SV Sandhausen II.
With the introduction of the Regionalligas in 1994 and the 3. Liga in 2008 as the new third tier, below the 2. Bundesliga, all leagues below dropped one tier. In 2012, the number of Regionalligas was increased from three to five with all Regionalliga Süd clubs except the Bavarian ones entering the new Regionalliga Südwest.