After a shaky start financially, the club advanced steadily through the lower leagues until it earned promotion to the Bezirksliga Rhein-Saar in 1931, but only played for a single season at that level before descending again. In 1943, it was merged with TSV Walldorf and VfB Wiesloch to form the wartime squad KSG Walldorf-Wiesloch. The combined squad was dissolved at the end of the conflict and SG Sandhausen was reestablished as an independent club late in 1945. A half dozen years later it re-claimed its original name. Sandhausen played football in the Landesliga or 2.Amateurliga until 1956 when it advanced to the 1.Amateurliga Nordbaden. In 1977, the team became German amateur vice-champion and advanced to the Oberliga Baden-Württemberg in 1978 where it consistently earned finishes in the upper half of the standings. Sandhausen claimed three Oberliga titles through the 1980s and the German Amateur Championship in 1993. It repeated as Oberliga champions in 1995 and 2000 and, with its latest title in 2007, advanced to the Regionalliga Süd (III).
Negotiations held in late 2005 and on into early 2006 to merge Sandhausen with TSG 1899 Hoffenheim and FC Astoria Walldorf to create FC Heidelberg 06 were abandoned due to resistance to the idea on the part of both Sandhausen and Walldorf, and the failure to agree on whether the new side's stadium should be located in Heidelberg.
The 2007–08 season was a success for the club, being in contention for 2nd Bundesliga promotion almost until the end of season and comfortably qualifying for the new 3. Liga. In 2012, the club won the 3. Liga and thus promotion to the 2. Bundesliga. The club finished its inaugural 2. Bundesliga season on a relegation rank but was saved when MSV Duisburg was refused a licence and played a much stronger 2013–14 campaign, finishing 12th.