This older castle had already changed hands several times as the Normans and Welsh fought for supremacy in these parts, and the small town of Builth had sprung up nearby.
As at Flint Castle begun at the same time, construction began with an enclosure of the site by a wooden palisade, in the case of Builth of some 270 yards in length.
Following Eubulus' death in 1335, and Alice's subsequent re-marriage, her new husband / rapist - Hugh de Freyne - was acknowledged by the King as lord of Builth.
In 1359, the Prince re-established the rental arrangement (still at 170 marks per half-year) with the Mortimer family, retaining the right to garrison the castle in times of war.
When it was attacked by Owain Glyndŵr's forces, during his revolt in the early 15th century, the garrison was under the command of Sir John Oldcastle (Shakespeare's Falstaff).
During the reign of Arthur's younger brother, King Henry VIII, the castle burnt down, and over subsequent years the ruins were gradually raided by the local population to provide building material for their properties.