The village toponym is derived from the Old English for "Hædde's cliff", referring to the position of the village on a cliff overlooking the Thames.
Hedsor House was the ancient seat of the de Hedsor family, who took their name from the village. They owned it at the time of the Domesday Book in 1086. The modern manor house was built in 1778, and stands on a hill in the village with a commanding view of the Thames and of Berkshire.
In 1583 Roland Hynd built a new Tudor manor house at Hedsor which was badly damaged by fire in 1795 and eventually demolished in 1865. At about this time the fourth Lord Boston commissioned the architect James Knowles[clarification needed] to design a new house at Hedsor. Hedsor House remained as the Boston family's country house until early in the 20th century, during which time Queen Victoria was said to be a frequent visitor to the house and its beautiful surrounding parkland.