It controlled a prominent position overlooking the Moray Firth and critical to the sea access to Inverness.

[1] By the thirteenth century, the castle was in the hands of the de Moravias of Petty and was where they ruled their vast tracts of land across Moray.

Andrew de Moravia mustered the men of Moray to join King John Balliol at Ormond Castle following his dishonour in front of Edward I of England, at the outset of Wars of Scottish Independence.

Following the fall of the Black Douglases after the Battle of Arkinholm in 1455 and the execution of the earl, his properties, like those of his brothers, was forfeit.

Angus, the "Red Douglas", was a kinsman and enemy of the Black Douglases, and became the new power from that mighty house.