The castle was built by William Rufus around 1092 within the old Roman fort of Verterae to protect a key route through the Pennine Mountains.

[1] The site protected the Stainmore Pass that stretched from the River Eden across the Pennines, and the Roman road connecting Carlisle and Ermine Street, a valuable trading route during the period.

[6][nb 1] The village of Church Brough was created alongside the castle at around the same time, in the form of a planned settlement, part of the Norman colonisation of the lowlands in the region.

[9] Brough, guarded by six knights, put up a strong resistance, but William took the outer defences and then besieged the keep, threatening to execute the garrison if the castle was not surrendered.

[10] The keep was set on fire, forcing the surrender of the garrison, including one knight who, according to the chronicler Jordan Fantosme, fought on first with spears and then wooden stakes, until finally overwhelmed.

[13] Robert enlarged the castle in order to exert his authority over the region, where he was competing for control with other members of his extended family.

[18][nb 2] His son's guardian, Hubert de Burgh appointed the Prior of Carlisle to run the estate and the castle was left to fall into ruin.

[21] The Cliffords successfully recombined the former Vieuxpont estates by 1333, and were able to controlled the Eden valley through their castles at Appleby, Brougham, Pendragon and Brough.

[24] Robert died fighting the Scots at the Battle of Bannockburn and the region around the castle was attacked in 1314 and 1319, causing significant damage to neighbouring Church Brough.

[25] Around this time the village of Market Brough was established along the road overlooked by the castle, in an attempt by the Cliffords to maximise the possibilities for profits from trade along the valley.

[34] Although Anne would have been familiar with contemporary styles, her restoration work was quite traditional in approach, drawing on existing northern castle architecture and deliberately trying to recreate 12th century features in the keep.

[35] As part of this work, new windows, a ground-floor entrance to the keep and new service accommodation was installed to allow her to live a late 17th-century lifestyle, and the castle had 24 fireplaces by 1665.

[5] The furnishings were sold in 1714 and in 1763 much of the stone from Clifford's Tower was plundered for use in the construction of Brough Mill; the castle was subsequently completely abandoned.

[32] Work to stabilise the ruins was carried out and the castle, as a listed building and scheduled monument, eventually passed into the control of English Heritage as a tourist attraction.