A stone castle was built on the site of an earlier defended position from around 1095 to 1125 by Guy de Balliol.

When John Balliol was deposed as King of Scotland in 1296 the castle was passed to the Bishop of Durham.

[4][5] During the rebellion of the Earls of Northumberland and Westmorland, in the reign of Elizabeth, the castle, which was then the property of the crown, was garrisoned by Sir George Bowes, of Streatlam.

After frequent grants and reversions, the castle, lands, and appurtenances, were purchased by Sir Henry Vane, an ancestor of the Duke of Cleveland, himself a Viscount Bernard.

[9] The remains of the medieval chapel of St Margaret in the outer ward are listed as Grade II.