It was built by the Normans to overlook the River Usk and its valley, and so guard against incursions into the lowland areas south and east of the town by the Welsh from the hills to the north and west.

[1] In the 1160s, Henry Fitzmiles, the son of Miles de Gloucester, 1st Earl of Hereford and lord of Abergavenny, was killed, reputedly by Seisyll ap Dyfnwal of Castell Arnallt.

Without a male heir, Henry Fitzmiles' estate and the lordship, which included lands in upper Gwent and Brecknockshire, as well as the Castle, passed to his daughter Bertha's husband, William de Braose.

Over Christmas 1175 De Braose called Seisyll and his son Geoffrey to his castle, together with other leaders from Gwent, supposedly as an act of reconciliation.

De Braose then had the men killed in the castle's great hall, in retribution for the death of Henry Fitzmiles.

[4] In the 13th and 14th centuries, the castle was expanded by the Hastings family, including the addition of western towers which provided residential chambers.

[3] This was the same time in which murage taxes were collected from local residents to fund the building of the town walls.

[1][5] In 1404, during the rebellion of Owain Glyndŵr, the town of Abergavenny was sacked and burned and the castle was besieged by Welsh forces.

[4] By the late 18th century, the ruins were starting to attract visitors seeking "picturesque" views, and walks were laid out within the castle walls.

[12][13] The castle grounds are designated Grade II on the Cadw/ICOMOS Register of Parks and Gardens of Special Historic Interest in Wales.