The original castle, constructed of wood, was founded in or soon after 1100 by Richard Fitz Pons, the owner of the nearby Herefordshire barony of Clifford, who was a supporter of Bernard of Neufmarché.

In 1144 Roger Fitzmiles, Earl of Hereford, detached the land surrounding Bronllys from the Lordship of Brecknock, and gave it to Richard's son, Walter de Clifford, who then rebuilt the castle in stone.

Maud (the daughter) died in 1311, and Edward II gave the castle to Rhys ap Hywel (a local resident descendant of the former kings of Brycheiniog)[1][2] as reward for loyalty to the crown.

After deposing Richard II, Henry used his royal authority and terminated the abeyance in Mary's favour, effectively transferring the lands to himself by jure uxoris in 1399.

His grandson, Edward Stafford, 3rd Duke of Buckingham, was suspected of plotting against King Henry VIII, and was subsequently executed for treason in 1521; Bronllys was forfeited to the crown, with which it has remained ever since.