Most of Northern Dyfed, except for lands owned by the Bishop of St. Davids—Dewisland—was taken by Martin de Turribus,[1] who became the first Marcher Lord of Kemes.

[5] None of this original castle survives, with the oldest remaining parts of the building thought to date to the late 13th century, after reconstruction following the attacks by the Llywelyns.

In 1326, two years after his similarly named father, the reigning Lord Martin (the Fitz having been dropped earlier in the previous century) died childless, and the Lordship was inherited by his sister, Joan.

The castle was temporarily transferred to the crown at the end of the century, in 1497, when the then Lord Audley, James, was executed for high treason and all his lands seized, but these were returned to his son John in 1534; the following year the status of Marcher Lordship was abolished by the Laws in Wales Acts.

[6] A three-storey private residence was built in 1859 on the site of the castle's gate-house, as part of renovations carried out by the owner at the time, Sir Thomas Lloyd, during which one of the flanking towers of the gatehouse was demolished.