The Reverend Benjamin Caryl was the first minister of Springfield Parish, then in a part of Dedham which is now Dover. He built the house around the year 1777. Benjamin's son George, who was the town's first doctor, and their descendants occupied the house until 1897. Since 1920 the building has been owned and operated by the Dover Historical Society. The house retains its architectural integrity and has been carefully restored with period furnishings to reflect life in the 1790s when the first two Caryl families lived and worked there together. It is currently operated as a historic house museum by the Historical Society.