It is conceivable that it was one of the two castles known to have been erected during the reign of William the Lion on the Black Isle, as royal fortalices.
By the thirteenth century, the castle was in the hands of the de Moravias of Petty and was where they ruled their vast tracts of land across Moray. Andrew de Moravia mustered the men of Moray to join King John Baliol at Ormond Castle following his dishonour in front of Edward I of England, at the outset of Wars of Scottish Independence.
The heiress to the de Moravias of Petty was Joanna of Bothwell, in 1362 she married Archibald the Grim, Lord of Galloway and later 3rd Earl of Douglas who claimed her estates and titles de jure uxoris (by right of marriage). Ormond Castle was the caput of the Barony of Ormonde, and was created into an Earldom in 1445 for Hugh Douglas, Earl of Ormonde, third son of James Douglas, 7th Earl of Douglas. Following the fall of the Black Douglases after the Battle of Arkinholm in 1455 and the execution of the earl, his properties, like those of his brothers, was forfeit. Ormond Castle was taken by the crown and regranted to George Douglas, 4th Earl of Angus. Angus, the "Red Douglas", was a kinsman and enemy of the Black Douglases, and became the new power from that mighty house. The title of Earl of Ormonde was recreated for the elder son of the 1st Marquess of Douglas in 1651, but became extinct upon the death of Archibald Douglas, 3rd Earl of Ormonde in 1715, who died of wounds following the Battle of Sheriffmuir.
In 1568 Ormond Castle and the lands of Suddie in Avoch were acquired under feu from the Earls of Angus and Ormonde, by Andrew Munro, 5th of Milntown, also known as Andrew Munro of Newmore during his father's lifetime.