Whilst early thirteenth-century fragments from the old hall still exist,[1] the palace, including the 25 metres (82 ft) great hall, was largely constructed under Bishop Henry de Gower, Bishop of St David's (1328–1347) and was used by high-ranking clergy.

[2] The palace was built in three stages and originally had over 20 rooms and featured fishponds, orchards, fruit and herb gardens and areas of parkland with grazing deer.

[3][4] The palace was surrendered to the crown in 1546 during the reign Henry VIII and granted to Richard Devereux and subsequently the Earls of Essex.

[1] Walter Devereux, 1st Earl of Essex lived in the palace as a child.

[5] The palace was sold to the Owens of Orielton in 1683, possibly due to damage in the English Civil War.