Construction was begun by the order of Tokugawa Ieyasu in 1608, and it was completed in six months. Ikeda Terumasa was in charge of the construction; the plan of the castle was done by Tōdō Takatora. It is said that 20 daimyōs were mobilized for the construction. The castle was so well designed that Ieyasu commanded that the tenshu, or keep, not be constructed because he feared that it would be used as a base against him.
Almost all of the buildings in the castle were destroyed after the Meiji Restoration, except for the Ōshoin (Grand Hall). However, the Ōshoin was destroyed during a firebombing air raid by American forces in 1944. It was reconstructed in 2000.
Media related to Sasayama Castle at Wikimedia Commons