The first incarnation of the fort was built between 1652-3 during the First Anglo-Dutch War. Little is known of the original structure and no trace of it has been found.
The second structure was built on the same site by Robert Mylne under the orders of Charles II at the start of the Second Anglo-Dutch War in 1665 at a cost of £28,000. It held off a Dutch fleet in 1667 which thought it was far more heavily manned and gunned than it actually was. In fact, the walls were unfinished and there were few guns. At the end of the war it was slighted when the government decided not to station a garrison in Lerwick, and it was unmanned when the Dutch burnt it in 1673 during the Third Anglo-Dutch War.
It was rebuilt in its current form in 1781 and named after the wife of George III but has never seen service during hostilities since then. It housed a garrison during the Napoleonic Wars and was later a base for the Royal Naval Reserve. From 1837-75 it was used as the town jail and courthouse and later a custom house and a coastguard station.
Due to land reclamation and subsequently erected docks and buildings in front of the fort, it no longer dominates the shoreline and buildings in close proximity means the overall shape can only be seen from the air.