A house was built on the site in 1798 when Reverend Nathaniel Thayer (1769-1840) constructed his parsonage in South Lancaster. His sons took down the original home and built the estate in 1846. It was enlarged and remodeled in 1902 by the noted architect and interior designer Ogden Codman to 42 rooms.
After being sold out of the Thayer family and having its furnishings sold at auction, the house was sold to Atlantic Union College in 1943 at a cost of $12,500. Today the college's Music Department is housed in this mansion, renamed the Thayer Performing Arts Center.