The land rises steeply from near the water's edge and the picturesque village is boxed in along the shoreline by the heights above. The small harbour itself is not a particularly good natural harbour and is protected from swells by a breakwater that was built with thousands of huge blocks of sandstone quarried out of the face of the mountain above. Simon's Town is now in effect a suburb of greater Cape Town. The Simon's Town railway station is the terminus of the Southern Line, a railway line that runs south from the central business district of Cape Town. In places the railway line hugs the steep eastern shore of False Bay quite spectacularly and in bad weather waves and foam from some heavy swells will fly up and wet the trains.
Boulders Beach is located a few kilometres to the south of Simon's Town, in the direction of Miller's Point. Here small coves with white sandy beaches and calm shallow water are interspersed between boulders of Cape granite. There has been a colony of African penguins at Boulders Beach since 1985. There is no record of the birds having lived here prior to that date, so their decision to settle in an area already well-utilized by humans is remarkable. There are only three penguin populations on the mainland in southern Africa; the others are close to Hermanus at Stoney Point and Betty's Bay.
In the last weeks of 1795 or the first weeks of 1796 the British built a round tower on a site that today falls within the Naval Base. Britain had just annexed the Dutch colony at the Cape of Good Hope and wanted to establish some defences to ward off possible Dutch or French attacks. The resulting tower was 26 feet high, had a base diameter of 42 feet, and walls 6 feet thick, though there is no evidence that the British ever installed the guns the tower was designed to hold. In front of the tower the British also constructed a battery that they did arm with cannons. The Martello Tower was used as a navigational beacon for ships entering Simon's Bay and was consequently white-washed in about 1843. The tower was restored in 1972 by the Simon's Town Historical Society (in conjunction with the South African Navy), and proclaimed a provincial heritage site; today it houses a small museum.