Mathew St. Clair Clark, a House of Representatives clerk, began the original brick building in 1836. Shortly afterwards, it was sold to the British government. Lord Alexander Ashburton took up residence in the house in 1842, the same year he and Daniel Webster negotiated the Webster-Ashburton Treaty in its parlor. Ashburton was succeeded by novelist Edward Bulwer-Lytton; both made changes to Clark's design. The house received its French Second Empire design in the 1870s. The architects for both the 1853 and c.1870 remodelings are unknown. In the early 1950s, the house became the parish house for nearby St. John's Church.