The Normans began to make incursions into South Wales from the late-1060s onwards, pushing westwards from their bases in recently occupied England.

[5] A local church, St Mary's, was built just to the east and a settlement grew up around the site; a rabbit warren was established in the nearby sand dunes.

[6] In the early 13th century, a simple stone hall, approximately 18.6 by 7.6 metres (61 by 25 ft), was built on the site of the older timber building, using red-purple sandstone with white limestone detailing.

[9] A gatehouse was built as the new entrance, with two half-circular towers that possibly imitated those of regional castles such as Caerphilly; it was weakly defended by a portcullis and a handful of arrow loops.

[11] The Braoses may have rebuilt Pennard as a replacement for their castle at nearby Penmaen which was abandoned at around the same time due to encroaching sand dunes.

[16] By 1922, concerns had grown about the condition of the castle and discussions took place between the Pennard Golf Club, who owned the site, the Royal Institution and the Cambrian Archaeological Association.