It originally stood on a small island in the Firth of Forth accessible only at low tide, and dates from around 1450, built as a secure residence by Sir David Stewart, who had been granted the Barony of Rosyth in 1428.
The original tower house (58 feet high) was enlarged and extended in the 16th and early 17th centuries.
It remained a Stewart residence until it was sold in the late seventeenth century to David Drummond of Invermay. It ultimately ended up in the possession of the Earl of Hopetoun and from the eighteenth century onward remained unoccupied. During this and later periods large parts of the stonework were re-used in other structures, and the later courtyard buildings were almost razed to the ground, leaving only the tower and north courtyard wall remaining significantly above ground-floor level.
It became Admiralty property in 1903 and as the result of land reclamation lost its waterfront position, becoming marooned within the dockyard. Although plans were made to restore and use the building, they came to nothing and the structure was made safe in its current condition. It passed into private hands when large tracts of the surrounding dockyard were sold.