The tower was built by the Earls of Lennox, who belonged to the Stewart family. Mary, Queen of Scots and Regent Morton visited the castle, while James VI used it as base for hunting. It was later acquired by George Heriot, the goldsmith and founder of George Heriot's School. The castle became so ruinous that at one stage it became a rock garden.
The castle was a rectangular tower. The only part remaining is the basement, the entrance of which is in the northeast corner, and the base of a turnpike stair. The north and west walls of the tower are about 8 metres (26 ft) high, while the other two sides are less than 2 metres (6.6 ft).
There was a tunnel from the building to Lymphoy House. The castle is on a promontory. There is a ditch to the south of this, about two metres deep, where an inner rampart may have been. It is thought that a barmkin once surrounded the site, while other buildings probably stood between the tower and the ditch.
The tower is said to be haunted by a white lady. The tradition is that she is the daughter-in-law of the family at Lymphoy House. After her husband died in battle she turned to his family, who had never approved of her, for help. When they rejected her she, and her child, died in the snow. A white lady has been seen gliding around the area several times since.