Hildasay has an area of 108 hectares (0.42 sq mi), and is 32 metres (105 ft) at its highest point. It consists of red-green granite (epidotic syenite) that was quarried for many years.
The south coast has two narrow inlets, Cusa Voe and Tangi Voe. "West", the larger of two lochs, has a single islet. A satellite island, Linga lies to the south east. A long line of skerries and holms lies to the north west.
The island possibly takes its name from a Norse war goddess.
Hildasay has been uninhabited since the late nineteenth century, but as late as 1891 had a population of 30. The island's former industries included curingherring and quarryinggranite. The remains of a railway line leading from the quarry to the harbour can still be seen.