Extensions were added in 1770 by General James Grant of the American Wars of Independence (whose ghost is said to haunt the castle) and in 1850 by the architect Thomas MacKenzie. Further extensions carried out in 1878 were mostly demolished during and modernisations enacted in 1965. It has been continuously occupied by the Russell and Macpherson-Grant families throughout its existence.
The castle grounds contain a 20th-century rock garden and a 17th-century dovecote. The rivers Spey and Avon flow through the grounds, offering excellent fishing. The famous Aberdeen Angus cattle herd resides in the castle estate.
Today, the castle is still occupied by Clare Nancy Russell and her family. Mrs Russell is the current Lord Lieutenant of Banffshire. It is open to tourists during the summer months and a number of workshops on its grounds are in active use.
It is said that the original intention was to build at a better site defensively, but when building commenced whatever was built in the day was thrown down at night. Eventually the laird, annoyed by the problem, heard a mysterious voice saying "Build in the cow-haughs, and you will meet with no interruptions. He did so, and there was no further problem with the building.
- "Scotland’s specters: Haunted castles, part 1 - Heritage". Scotsman.com. Retrieved 2012-05-07.
- Tranter, Nigel (1993)Tales and Traditions of Scottish Castles. Neil Wilson Publishing. ISBN 1-897784-13-9 p.15