The current building dates from c. 1507, though fortifications were present c. 1400. The castle has four storeys and a vaulted basement. It was built by the Arnot family who have records dating back to 1105. David Arnot of Fyfe (Fife) was one of 2000 noble landowners required to swear allegiance to King Edward I of England in 1296. Nicol Arnot Arnot was a loyal supporter of King Robert the Bruce. Robert Arnot was killed in the battle of Flodden in 1514. The Arnots abandoned the old tower around 1700. The castle is a scheduled monument.
In 1760 local poet Michael Bruce wrote a poem about the true story of a love affair between an Arnot daughter and a Balfour of nearby Burleigh Castle. The families were in a feud, and it is believed the daughter of Arnot eloped to Burleigh Castle.