In 1811, Lyman Harding, the first Attorney General of Mississippi, hired Levi Weeks, a young self-proclaimed architect, to design and build the mansion. The original building was 60 feet (18 m) by 45 feet (14 m) with a two story portico supported by four columns and includes an unusual floating spiral staircase between the first and second floors. Levi Weeks designed the house to be, in his words, the "most magnificent building in the state."
After Harding died in 1820, the building was bought by Dr. Stephen Duncan, who expanded the house to include two symmetrical wings, greatly expanding interior space. Duncan later abandoned the building amidst growing secession tensions just before the American Civil War, and the house was placed in the care of his descendants.
The building and 222 acres (90 ha) was sold by the family to the city of Natchez in 1911 and is now a historic house museum in a public park.