The manor of Clovelly was for over 600 years owned by the Cary family but in 1738 the estate was sold to locally born Lincolns Inn lawyer Zachary Hamlyn. He replaced the old manor house with the new and present mansion in 1740. On his death he left the property to his nephew James Hammnet who changed his name to Hamlyn. He married Arabella Williams, a Carmarthenshire heiress, and was created a baronet in 1795 (see Hamlyn-Williams baronets). In that year he carried out substantial improvements to the house.
The 4,600-acre  estate passed on the death of the 3rd and last baronet in 1861 to his son Neville and then following his early death to the baronet's eldest daughter Susan Hester Hamlyn-Williams who married Henry Fane, creating the Hamlyn-Fane family.
The house and estate remain in the family and are managed by the Hon. John Rous (born 1950), great-great-grandson of Susan Hester Hamlyn-Fane, great-grandson of Prime Minister Herbert Asquith and son of the 5th Earl of Stradbroke.
The gardens and parts of the estate, but not the house, are open to the public.
- Heritage Gateway: architectural description of listed building
- The Acre-ocracy of England 1876 John Bateman