It was built in ashlar in 1870 to a rambling plan in the Scots baronial style. Although roofs have fallen in the walls are sound and consist of two storey accommodation wings dominated by a four storey keep and a connected five storey tower. It stands in a 2000-acre moorland estate.
Kildalton was designed in 1867 by John Burnet and built by 1870 for the Islay distiller and MP John Ramsay. Ramsay had taken out a lease on the rundown Port Ellen distillery and made a success of it, improving Port Ellen's dock facilities and pioneering the export of whisky to America. The house he built stood in a moorland estate of 54,000 acres. Ramsay was a Deputy Lieutenant and a J.P. for Argyllshire and elected MP for Stirling Burghs in 1868 and Falkirk Burghs in 1874. He died in 1892 and the distillery continued under his wife and son, but in 1920 was sold to the Distillers Company.
Kildalton was sold in 1922 to John Talbot Clifton, a well-to-do landowner from Lytham, Lancashire who was passionately fond of shooting and foreign travel. He had travelled in Canada, Siberia, Burma, Malaya, Indonesia, Africa and South America, and was known for shooting wild animals and eating them. Some of the animals he shot were species new to science and were named after him, such as a type of wild Siberian sheep (Clifton's bighorn) and a Canadian marmot. He once dined on mammoth recovered frozen from the Arctic permafrost. He was married to the equally colourful Violet Beauclerk. His last venture was an expedition with his wife to Timbuktu, but he was taken ill en route and died in 1928 in the Canary Islands on the way home. His wife had his body embalmed and escorted it back to Scotland, where it was buried on Islay at Cnoc Rhaonastil. Violet then left the island to live at their Lytham house and Kildalton was neglected. Gradually the building deteriorated to the stage where floors and roofs collapsed, leaving only the shell standing. It was put on the Buildings at Risk Register for Scotland.
The current owners of Kildalton are George and Fiona Middleton, who bought 2000 acres including the ruinous castle structure. Various attempts to achieve the renovation of the castle have failed.