The mansion was built between 1752 and 1755 by a local nobleman, Anton Gaisruck, who had bought a decaying late medieval hunting mansion called Brutnberg (known locally as Plumberk) that stood on the same location. He used some elements from the old Celje Castle for its construction, causing further deterioration of this important historical site. In the mid-19th century, the mansion was bought by the Styrian nobleman Joseph Ludwig (or Jožef Ludvik) Hausmann, father of Franciska Romana "Fany" Hausmann (1818–1853), generally regarded as the first Slovenian female poet. He is also notable for introducing the cultivation of hops to Lower Styria, which launched a flourishing brewing industry in the Savinja Valley between Celje, Žalec, and Laško.
In the late 1920s, the mansion was bought by the local authorities and in the 1930s it was transformed into a mental hospital run by the Slovenian regional government. Most of the costly interior decorations were removed from the manor and placed in the National Museum of Slovenia. When the armed forces of Nazi Germany occupied Slovenia and Lower Styria in April 1941, the 390 patients of the institution were transported to Germany and killed by members of the Wehrmacht. After the Second World War, a memorial plaque was placed on the facade of the building to commemorate this tragic event.