Following the time of the Norman Conquest the manor was granted to John de Bredwardine.

In the following century it was held by Hugh de Lacy.

[citation needed] Bredwardine Castle was rebuilt as a fortress during the wars of Stephen and Maud but then dismantled in the reign of Henry II or Henry III.

In the middle of the 15th century the castle was described as being a "waste site with no annual value".

Roger Vaughan was the son in law of Dafydd Gam and converted the castle and manor into a multi-gabled house.