The Norman motte and bailey castle stands on a small prominence, a somewhat elliptical mound,[2] above the River Wansbeck.

The selected building site allowed for the natural hill to be scarped and ditched, producing the motte.

[1] In 1264, the stone castle was held by the third Roger Bertram, but in that year, it was seized from him and committed to the custody of William de Valence, 1st Earl of Pembroke, King Henry's half-brother.

It was certainly ruinous by 1323, as records of an inquest held that year after the death of its then owner Aymer de Valence, Earl of Pembroke, state Mitford Castle to be " entirely destroyed and burnt.

"[2] At the time of his death in 1335, Mitford Castle had been seized from its then holder, David II Strathbogie, Earl of Atholl, 2nd baron.

[9] Remains of a 12–13th century east curtain wall of squared stone include a gateway to a barmkin (a defensive enclosure), mural chambers, a garderobe (a toilet), and a round arch.

[10] This east curtain wall area is flanked by a semicircular breastwork, the strongest part of the building.

[2] The west curtain wall and structures are also of the 12–13th century and squared stone, with different builds and masonry types found across three different sections.