It was built around 1850 for Charles Wilsone Brown, a property developer who had plans to develop the land around Wemyss Bay, and who by 1855 had increased the number of villas from four to thirty-six. These villas earned the nickname 'Little Glasgow' because they were let to wealthy Glasgow merchants.
Wilsone Brown sold the mansion to Sir John Burns (later Baron Inverclyde) in 1860. Burns commissioned the architect Robert William Billings to remodel the house in the Scottish baronial style, expanding the original structure by adding a new floor, new wings and a clock tower to the south-east corner.
Castle Wemyss became a fashionable destination for many well-known visitors, including Lord Shaftesbury, Anthony Trollope, General Sherman, Henry Morton Stanley, Peter II of Yugoslavia, Emperor Haile Selassie and members of the British royal family.
It is reputed that Trollope wrote part of Barchester Towers whilst at Wemyss Bay, and that 'Portray Castle' in The Eustace Diamonds was based on Castle Wemyss. Whether this is true or not, Trollope places Portray in a similar geographical location, with a description which is very like that of the castle and its grounds. Trollope did however include the real Castle Wemyss in his travelogue How the 'Mastiffs' Went to Iceland, a record of a trip from the Clyde to Iceland in June and July 1878.
The house remained in the Burns family until the death of Alan, 4th Baron Inverclyde, in 1957. None of his family was able to afford the cost of maintaining such a large property and it was sold to a developer. Inverkip Power Station was built on the northern part of the estate grounds.
Greenfield legislation meant that whilst much of the larger estate was developed as Wemyss Bay grew in the 1960s and 1970s, the house itself was left untouched. Gradually falling into decay (and subsequently de-roofed to avoid payment of housing rates) it was not until 1984 that it was finally demolished.
In the 1990s the grounds themselves were finally redeveloped, and all that now remains of the house is a broken flight of stone steps and a flagpole. An established monkey puzzle tree marking the entrance to the drive had to be removed for safety reasons. However, the locally formed Woodlands Trust have started to reclaim the Victorian fernery from the rhododendrons. Formal paths have now been established in the woods which are immediately adjacent to the site of the former castle.
- Smart, Walter (1968) Skelmorlie, Sklemorlie and Wemyss Bay Community Centre
- Trollope, Anthony, The Eustace Diamonds, Chapter XXI: Ianthe's Soul
- Digitized version of How the 'Mastiffs' Went to Iceland at The Internet Archive