He demolished the existing building and built the current house, designed by the leading architect Sir Jeffry Wyatville, 700 yards (640 m) to the south-west above the original (begun 1827, completed 1834).

[1] Wyatville was simultaneously occupied in the extensive remodelling of Windsor Castle for King George IV, and subsequently King William lV, as well as the building of a remarkably similar property, Lilleshall Hall in Shropshire, for Cawdor's cousin, George Leveson-Gower, nephew of the Duke of Sutherland.

Correspondence exists between the two families relating to the huge costs of construction of both Golden Grove and Lilleshall, and the problems in bringing them to completion created by Wyatville's preoccupation with the complexities and demands of his new schemes for the king.

Subsequently, Carmarthenshire County Council used it as offices, and Coleg Sir Gâr operated it as an agricultural college until 2003.

[2] Plans were made to convert the building to a hotel, to flats, and to a convalescent home, but the building remained unused and deteriorating until 2011, when the council sold the house and park to the Golden Grove Trust, formed to make the house a destination for art and cultural activities and restore the park.