Colonel Seaborn Jones drew all the plans for his home and called it El Dorado, land of beauty. The house was begun in 1828 and completed in 1833. The materials used to build this house, with the exception of the marble and the mahogany, were taken from the property itself. A small lake now fills the place where the clay was removed. To this El Dorado, Colonel Seaborn Jones brought his wife and children, a daughter and a son, in 1833. In 1833, Henry L. Benning, an aspiring young lawyer, wrote a friend: "Above all things (I advise if you desire ease and happiness) marry. Marry a lady of accomplishment, i.e. worth $100,000. It will be better than quibbling. I am anxious to experiment at least." In 1839, Benning, for whom Fort Benning is named, married Colonel Seaborn Jones's daughter.
At approximately 2:00 p.m. on October 5, 2011, a fire broke out in the basement wine cellar of St. Elmo. Apparently started by a single candle, the fire did extensive damage to the structure, but it is believed to be repairable.