The old castle, Schloss Ilsenburg, lying on a high crag above the town, was originally an imperial stronghold and probably built by King Henry I. In 995 Emperor Otto III resided in Elysynaburg, which Henry II bestowed in 1003 upon the Bishop of Halberstadt, who converted it into a Benedictine monastery. The school attached to it enjoyed a great reputation towards the end of the 11th century. The abbey was finally devastated during the German Peasants' War in 1525.
After the Reformation the castle passed to the counts of Wernigerode, who restored it and made it their residence until 1710. Higher still, on the edge of the plateau rises the Ilsenstein, a granite peak standing about 500 ft (150 m) above the valley, crowned by an iron cross erected by Count Anton von Stolberg-Wernigerode in memory of his friends who fell in the Napoleonic Wars of 1813-1815. Around this rock cluster numerous legends.