Raudonė has been traditionally identified as the location of Bayerburg, a castle of the Teutonic Order during the Lithuanian Crusade. A local legend claims that when Gediminas, Grand Duke of Lithuania, attacked the fortress in 1337, he was mortally wounded under the oak tree now known as the Gediminas Oak. However, there is no proof, either from historical record or archaeological research, that a castle stood there before the 16th century.
In the 16th century, Raudonė Castle belonged to King Sigismund II Augustus of Poland. A new castle was built on the ruins of the old one by a German knight, Hieronymus Krispin-Kirschenstein. The castle has since been rebuilt many times. The 18th century Polish owners of the Rudonė estate, the family Olędzki (Olendzki) h. Rawicz (members of szlachta, general sejm and senate) commissioned Laurynas Gucevičius with a renovation of the castle. The next owner, the Russian Prince Platon Zubov, acquired the estate in the first half of the 19th century and his family transformed the castle yet again. Their architect was Cesare Anichini (Cezaris Anikinis). Today the building is an example of 19th century neo-Gothic architecture. Its last private owners were Sophia Waxell (a Zubov) and her Portuguese husband from Madeira, José Carlos de Faria e Castro.
The original castle of Raudonė is the setting of an East Prussianlegend known as Die weiße Jungfrau der Bayerburg ("The White Maiden of Bayerburg").