An early reference to a castle at Kenfig can be found in 1080, when Iestyn ap Gwrgan was said to have refortified it, but probably this was a different structure to that raised alongside the town that developed there in the mid-12th century.
In its day, it was an important Norman stronghold and was built by Robert, Earl of Gloucester, in the early 12th century.
It was a tall, elegant structure with buttresses of dressed stone at each corner and the centre of each side, as well as a hall and offices.
The ramparts were removed to make the court more level and a curtain wall was erected, with a large gatehouse leading to the borough.
John Leland wrote in about 1539, "There is a little village on the e[a]st side of Kenfig, and a castle, booth in ruine and almost shokid (choked) and devourid with the sandes that the Severn Se ther castiith up".