The club has roots that go back to a cricket club founded in 1895. As they broadened their interests they came to be called Sport Club Erfurt. The club was a founding member of the German Football Association in 1900 and in 1904 they joined the Verband Mitteldeutscher Ballspielvereine (Central German Football League). The side won the league championship in 1908–09 and advanced as far as the semi final of the national round where they lost to the eventual champion. While Erfurt did manage to play for a number of seasons in the premier level Gauliga Mitte, formed after 1933, they failed to earn any honours.
In the aftermath of World War II the Allies banned all organizations, including sport and football clubs. In 1946, the Soviet occupation authorities permitted the organization of five district sports clubs in Erfurt. SG Erfurt West encompassed the area of the city once served by SC Erfurt 1895 and VfB Erfurt and drew footballers who had played for these clubs. Success came quickly with an appearance in the 1948 Thüringer final, followed by a title in 1949. A quick series of name changes went hand-in-hand with a series of failed cup and final appearances: as Fortuna Efurt in 1949, KWU Erfurt in 1950, and BSG Turbine Erfurt in 1951. In 1954 and 1955, Turbine captured consecutive East German national titles, but then slipped back into the pack and out of tier I for the first time in 1959. The team was up and down between the first and second divisions through the 60s, being relegated three times, but always winning immediate promotion. Like other East German clubs at the time they suffered as the best players were routinely plucked to play for favoured clubs with politically powerful sponsors.
East German football underwent major changes in 1965 with the creation of "pure" football clubs in the place of broadly generalised sports clubs. The number one football sides of SC Turbine Erfurt and BSG Optima Erfurt were merged in 1966 and revived the name FC Rot-Weiß Erfurt, while the more junior sides stayed with their original clubs. Rot-Weiß was unusual for its time in that the club did not have a name that honoured some socialist virtue. In 1980, Rot-Weiß appeared in the East German Cup final, losing to Carl Zeiss Jena (3–1).
German re-unification and the merger of the leagues of East and West Germany in the early 90's brought exciting times to Rot-Weiß. A third place finish in the NOFV-Oberliga in 1990–91 earned them a spot in the 2. Bundesliga for the next season, as well as a turn in the UEFA Cup 1991–92 where they went out against eventual winner AFC Ajax Amsterdam in the second round. That effort made them the last side to appear internationally for East Germany.
Through the 90s and into the new millennium, Rot-Weiß remained a tier three side. They had a close call in 2001 when they avoided relegation only because SSV Ulm 1846 was denied a license because of its financial difficulties. During this time the club went to the league final seven times and came away as Thuringian champion each time. Unfortunately, they were never able to make it past this point and were always subsequently put out in the first round of the German Cup. In 2004, the club made its way back into 2.Bundesliga, but finished dead last and so currently plays in Regionalliga Nord (III). In 2008, Erfurt finished in seventh place in Regionalliga Nord and so qualified for the new 3. Liga. it has played at this level since with two fifth places in 2011 and 2012 as its best results.
The club's reserve team, FC Rot-Weiß Erfurt II, currently plays in the tier five NOFV-Oberliga Süd. It first played at this level from 2005 to 2007, and again since 2008 with a third place in 2012 as its best result.