The present castle is on the site of a 14th-century castle. The castle belonged to the Lindsays of Crawford. It was designed to control Highland marauders. It was here that David Lindsay, 9th Earl of Crawford died in 1558. The present castle was built in the 16th century, and heightened in the early 17th century. The castle was abandoned in 1803.
The 16th-century castle was a three-storey structure, having a corbelled parapet and parapet walk. The additions were another storey and a garret, and a two-storey angle-tower. The castle walls have rounded corners. Two massive chimney-stacks have window-openings giving the garret light.
The entrance, at first floor level, was reached by a movable timber bridge or stair. The entrance, a rounded arch, which still has an iron yett, led to the hall, to which a small room is attached. A wheel stair was the only access to the vaulted basement. The turnpike stair by which access could be gained to the upper floors, which were also subdivided, no longer exists.
There are foundations of outbuildings to the east and south of the tower; material known to have been robbed from the site to build the parish church probably came from here.
- Lindsay, Maurice (1986) The Castles of Scotland. Constable. ISBN 0-09-473430-5 p.292
- "Invermark Castle". Canmore. Retrieved 2013-09-04.
- "Invermark Castle". British Listed Buildings. Retrieved 2013-09-18.