The first record of a settlement, located on the river Mulde, called Cholidistcha dates to the year 1046. In 1083, Henry IV recommended that MarkgrafWiprecht of Groitzsch build a castle on the cliff above the river. In the 12th century, houses were built around the marketplace and the St. Nicholas church was built. In 1265, town rights were granted by the ruler. In 1504, the baker accidentally set Colditz on fire, and the city hall, church, castle and a large part of the town went up in flames. In 1506, reconstruction began and new buildings were raised around the rear castle courtyard.
In the 17th century, a textile and weaving industry developed. In the 18th century, clay from the Colditz area started to be used in the Meissen porcelain factory that was established in 1710 by the Elector of Saxony, Augustus the Strong. In 1804 a ceramics factory was established in Colditz by Thomsberger & Hermann.
In World War II, the town did not suffer any damage. The town became headquarters for the military personnel guarding the prisoner of war camp for officers, Oflag IV-C, that had been established in the castle. On 14April 1945, the U.S. Army entered the town and freed the prisoners. However, under agreements signed at the Yalta Conference, the Americans withdrew and were replaced by Soviet occupation forces late in June 1945. As a result, Colditz and the entire state of Saxony became part of East Germany. In 1958, a factory manufacturing porcelain was established 
Since German reunification in 1990, efforts have been made to increase visits by tourists. The castle was restored and has become a much visited museum. The great flood of August 2002 caused some damage to the old town, but it has since been restored.