A fortified Norman castle with stone walls was built on the site possibly by William Fitz Osbern or his son Roger de Breteuil, Earl of Hereford in the period 1070 to 1075.
Historically the castle remained a part of Brecon or Brycheiniog barony until 1207 when King John of England granted it to Peter FitzHerbert.
The castle was sacked by Prince Llywelyn ab Iorwerth in October 1233 and subsequently refortified by King Henry III of England before being returned to Peter Fitz Herbert.
What remains now are crumbling walls mainly covered with earth and the outlines of ditches and ramparts from the original Iron Age fortifications, commanding extensive views up into the Black Mountains and over Talgarth towards Brecon.
The only extant building is section of a gatehouse next to a postern gate at the northern end of the outer ward.