The portrait painter George Romney bought the original house in 1796, at the age of 61, having previously lived at 32 Cavendish Square in London.

"[3] Romney was instead persuaded to buy the house on Holly Bush Hill and add a "gallery and painting-room at small expense.

[3] One of the pictures damaged was Nature Unveiling Herself to Shakespeare, which had been mentioned in poetry by Helen Maria Williams.

[4] The arcade extension later sold for £357 (equivalent to £46,701 in 2023)[4] and the original house for close to the £700 purchase price.

[5] The English section of an international writers organisation called "For Intellectual Liberty" was hosted by the Williams-Ellises' at the house[7] and Williams-Ellis entertained a number of prominent writers there including Cecil Day-Lewis, Goronwy Rees and Rose Macaulay.

[11] Romney's stables were extended in the early nineteenth century when the house was converted for use as an assembly rooms.

[12] When Clough Williams-Ellis bought the house in the early twentieth century, further substantial alterations were undertaken.