The house was built for Thomas Halsey (1655-1715), a Member of Parliament, in the 17th century. The family moved out in 1774 when Gaddesden Place was completed. The Golden Parsonage then passed down the Halsey family.
When a young boy, Thomas Frederick Halsey lost his parents and younger brother in a ship accident while on a holiday trip in the Mediterranean sea and was looked after by his maternal grandfather, General Frederick Johnston of Hilton, the last of the ancient Scottish family. Frederick's letters and diaries still depict his sufferings upon losing his entire family at a very young age.
Inheriting the estate from his grandmother Sarah Moore Halsey in 1869, Fredrick took it upon himself to add farm buildings, houses, cottages and also introduced a billiard room at the Golden Parsonage and the conservatory at Gaddesden Place. The house became a school in 1875 and remained in that use until it returned to residential use in 1935.
- "History". The Gaddesden Estate. Retrieved 10 August 2013.
- "'Parishes: Great Gaddesden', A History of the County of Hertford: volume 2 (1908), pp. 201-207". Retrieved 10 August 2013.
- "Golden Parsonage". Pastscape. Archived from the original on 11 August 2013. Retrieved 10 August 2013.