Over the next few decades, it passed back and forth between several owners, as Hubert, the rival de Braose family, and the Crown took control of the property.

During this period, White Castle was substantially rebuilt, with stone curtain walls, mural towers and gatehouses, forming what the historian Paul Remfry considers to be "a masterpiece of military engineering".

Edward I's conquest of Wales in 1282 removed much of White Castle's military utility, and by the 16th century it had fallen into disuse and ruin.

[6] The first castle on the site was built from earth and timber, with three large earthworks forming an inner and outer ward, and a hornwork protecting the main entrance to the south.

[14] At this time, White Castle was primarily a military fortification, holding a garrison and stores of arrows and crossbow bolts.

[15] It was relatively exposed to the elements and had, at best, only basic accommodation; the historian Cathcart King describes the conditions in the castle as likely to have been "miserable", "squalid" and "unpleasant".

[17] Once released, Hubert regained his grip on power, becoming the royal justiciar and being made the Earl of Kent, before finally recovering the Three Castles in 1219 during the reign of King Henry III.

[7] The Welsh threat persisted, and in 1262 the castle was readied in response to Prince Llywelyn ap Gruffudd's attack on Abergavenny in 1262; commanded by its constable Gilbert Talbot, Grosmont was ordered to be garrisoned "by every man, and at whatever cost".

[24] Evidence was given to the Royal Commission on Ancient Monuments in Wales and Monmouthshire in 1909, stating that Sir Henry had taken steps to strip the castle of ivy and that it was now in a good condition; the site was apparently looked after by an old woman, who charged visitors for entry.

[24] In the 21st century, White Castle is managed by the Welsh heritage agency Cadw and is protected under UK law as a grade I listed building.

[2] The outer ward is 320 by 170 feet (98 by 52 m) across, accessed by a gatehouse on the eastern edge and defended by a stone curtain wall, a dry ditch and four mural towers.

[35] Stretching eastwards from the gatehouse are the castle's hall, the constable's living quarters, the chapel – partially contained in one of the towers- the remains of the earlier keep, service buildings and the kitchen block.

[37] A postern gate in the inner ward leads through to the southern hornwork, which would originally have been linked by a wooden bridge, protected by timber defences and towers, with later stone additions, of which only traces remain.