After Oengus' defeat and death in battle, David installed Freskin, a nobleman probably of Flemish origin, as his chief agent in Moray,[2] and he was probably the first to build a castle at Duffus.

[3] Undoubtedly, King David, himself a Scoto-Norman monarch with extensive estates in northern England and Normandy, granted lands to many nobles from Flanders as well as Normans.

[5] It was Freskin who built the great earthwork and timber motte-and-bailey castle in c. 1140 on boggy ground in the Laich of Moray.

The bailey contained the buildings necessary to sustain its inhabitants – brew and bake houses, workshops and stables – as well as the living accommodation.

The castle may have been destroyed in 1297[7] during the First War of Scottish Independence,[8] and it might have suffered further during King Robert the Bruce's campaign in Moray in 1306.

In 1270, the castle passed into the ownership of Sir Reginald le Chen (d.1312) by marriage to the heiress Mary de Moravia, a descendant of Freskin.

In 1305, it was recorded that Reginald le Chen received a grant from King Edward I of England of 200 oaks from the royal forests of Longmorn and Darnaway "to build his manor of Dufhous",[9] demonstrating that a large construction project was being carried out.

In 1689, John Graham, 1st Viscount of Dundee was a guest of Lord Duffus just before the Battle of Killiecrankie, and would be one of the last important visitors before the castle's abandonment.