The large Georgian house was built in 1800 by local doctor Robert Darwin. His son Charles Darwin was born there on 12 February 1809. Robert Darwin died in the house on 13 November 1848, followed by his wife Catherine and unmarried daughter Susan in 1866. The Mount was then put up for auction, after the three surviving Darwin children, Ras, Charles, and Caroline Wedgwood had taken what possessions they wished.
In 1996 The Mount Residents' Group was formed, motivated by concern over a proposed development of 18 houses on land at Rowton Lodge, a Grade II listed house in the middle of The Mount. Due to pressure from the group, the development was cancelled.