The Normans built a castle in the current location in the early twelfth century and this was rebuilt in stone.

A Norman knight, Richard Fitz Pons, received the lordship of Cantref Bychan in 1116 and he probably began construction of a motte-and-bailey castle in 1116.

King Henry II of England spent a great deal of money repairing the castle in 1159–62, but the Welsh captured it regardless.

It was briefly recaptured by Welsh forces under Llywelyn ap Gruffudd five years later.

King Henry IV of England visited the castle in 1400 and it was besieged during the Owain Glyndŵr rebellion three years later.