The house was added to the National Register of Historic Places not only for its architecture, but for its family's importance in the founding and development of Paintsville. In 1843, John Stafford, the original owner of the house, helped establish the city of Paintsville. Then in the 1930s, the Stafford family sold most of their 1,000 acres (400 ha) farm to the city, doubling it in size.
The rear part of the home was built circa 1843, while the front part was built circa 1888, making it the oldest house in Johnson County. Although the home was originally built for John Stafford, it was named for his son who accumulated his father's property after he died in 1869. Francis Stafford also built the main part of the house.
In 1861 a holly tree was planted in front of the house and it still stands today.
Since 1979, after the death of May Stafford, the daughter of Francis M. Stafford, the house has remained vacant. In 2003, the state of Kentucky offered a $200,000 grant to the city of Paintsville in order to help with the purchase and restoration of the historic home. On July 12, 2003, the Paintsville City Council declined the grant.
In April 2010, a resident from nearby Floyd County purchased the home. After two years of renovations, the Stafford House opened for tours in October 2012.