Four stone Martello towers were built to strengthen Kingston's defences in 1846 during the Oregon boundary dispute between the United States and Britain. The towers were meant to protect the shipyard and the entrances to the Rideau Canal and St. Lawrence River, from possible United States aggression. Fort Frederick was one of these, and was built on the site of the original fort. Fort Frederick was abandoned in 1870. Fort Frederick is the RMC cadets' outdoor relaxed area, where all cadets are considered equal in rank, headress may be removed, and cadets have free rein to relax. The Martello tower houses the RMC Museum.
While Archibald Macdonnell was Commandant of RMC, between 1909–19, the upper floors of Fort Frederick were used as a common room. After the seniors rolled cannonballs down the common room stairs, the floor was reallocated as a recruit haven. The college's class of 1931 gifted Fort Frederick with wooden gates and a plaque in 1971 in remembrance of the days when Fort Frederick was a recruit refuge.
On 28 June 1985 Canada Post issued 'Fort Frederick, Ont.' one of the 20 stamps in the “Forts Across Canada Series” (1983 and 1985). The stamps are perforated 12Â½ x 13 and were printed by Ashton-Potter Limited based on the designs by Rolf P. Harder.