The castle was built between two arms of the River Würm for Duke Albrecht III, Duke of Bavaria in 1438–39 as a hunting-lodge, replacing an older castle burned down in war. The origin of this castle is a moated castle of the 13th Century. The core of this castle was a residential tower, the remains of which were uncovered in 1981. The fortress was first mentioned in writing only in 1432.
Albert's son, Duke Sigismund of Bavaria, ordered extensions of the castle beginning in 1488; he died here in 1501. The main building became derelict during the Thirty Years War, but was rebuilt in 1680–81. The castle is still surrounded by a ring wall with three towers and a gate tower. The defensive character of the castle, however, was with the reconstruction in 17th Century significantly reduced. The plant was already at that time no longer defensible.
Sigismund of Bavaria also ordered the construction of the palace chapel, a splendid masterpiece of late Gothic style which still has preserved its stained-glass windows, along with the altars with three paintings created in 1491 by Jan Polack. The cycle of the statues of the apostles on the side walls was built around 1490/95. The executive master is controversial, which is why the statues are assigned to the "Master of the Blutenburg apostles." The apostles can not always be clearly identified as the attributes were partially reversed or lost. Appendant to these figures the "Man of Sorrows" and the distinguished "Mother of God" (Blutenburger Madonna) were erected in the choir on crest consoles that match those of the apostles.