The castle was built in 1775 for Henry Disney Roebuck from designs by John Carter in the shape of the "clubs" symbol used in playing cards (♣).

[3] In 1810, the castle was bought by one of the Conollys of Castletown House in County Kildare, who added the porch (said to give the clubs symbol its stalk) and built the nearby stables and chapel, known as the priory.

[5] Soon after 1810 Kingham Field, which was part of the estate, was operating as a stone quarry similar to the nearby Combe Down and Bathampton Down Mines.

In April 1814, Smith mortgaged the remainder of his estate to Charles Conolly who then controlled the railway and probably extended it to his Vinegar Down Quarry.

[11] The castle grounds also include the Grade II listed archway with lodge and screen wall, [12] a pair of gatepiers and gates,[13] and the Grade II* listed group of the stables, old chapel, walls enclosing the stable yard, coach house and greenhouse.