Pieter Lefferts built the house in 1783 and it was located on Flatbush Avenue near Maple Street. Upon Pieter's death, the house passed to his son John, and then John's daughter Gertrude Lefferts Vanderbilt, who wrote about her family, her community and the house in her 1881 book, The Social History of Flatbush. The house remained in the Lefferts family for at least four generations.
Development was threatening Brooklyn's rural setting by the end of the 19th Century and in 1917, John Lefferts' estate offered the family's home to the City of New York with the condition that the house be moved onto city property as a means of protection and historic preservation. The house was moved six blocks to Prospect Park in 1918 and in 1920, the Fort Greene chapter of the Daughters of the American Revolution opened the house as a museum.