Dating from 1790, the castle was built by William Jones, owner of the Clytha Park estate as a memorial to his wife, Elizabeth, who died in 1787.

The folly has views towards the Sugar Loaf and Skirrid mountains on the easternmost edge of the Brecon Beacons National Park.

Described by the architectural historian John Newman as one of the two "outstanding examples of late eighteenth century fanciful Gothic in the county", Clytha Castle is a Grade I listed building.

[5] William Jones engaged John Davenport, an architect based in Shrewsbury to design the castle, built "for the purpose of relieving a mind sincerely afflicted by the loss of a most excellent wife".

[4] The folly experts Gwyn Headley and Wim Meulenkamp describe the architectural style deployed at Clytha as "a late fling of Strawberry Hill Gothick” and the building as "the Taj Mahal of Wales".