[4] In 1345, at the death of Reginald le Chen, Baron Inverugie, the estate of Inverugie had passed to the Keith Earl Marischals, who had their main seat at the coastal fortress of Dunnottar Castle (via marriage of Edward Keith and the heiress Marjory, daughter of Reginald le Chen and Helen de Strathearn).

Around 1660 the Keiths built what is the current, but now ruined, castle, lying south of the original wooden motte.

[5] In the 19th century, an oak heraldry shield was found in a local cottage with the arms of William Keith, 7th Earl Marischal and its date was carved as 1660.

The estate factor, William Ainslie, probably acting under instruction from the Laird at that time, arranged to have much of what was left of the ruins blown up, weakening the remaining structure.

[7] Charles McKean described the castle as "a splendid double-courtyard Renaissance chateau [consisting] of a four-storey block with circular angle towers and a stair turret".