They had extensive coal-mining interests across the South Wales Coalfield, particularly the colliery at Cymmer, Rhondda Cynon Taf, and were closely involved in the development of the Barry Railway Company and the associated docks.

Their growing prosperity enabled them to invest in land, including a site within the ancient Cathedral-city of Llandaff, between roads leading to Fairwater and Ely.

In the 1870s James Harvey commissioned George Robinson and Edwin Seward to change the appearance of the building into the Gothic Revival architecture-led works reminiscent of those of Lord Bute's architect William Burges on Cardiff Castle.

[6] The second development in 1906 - the height of the family's fortune - was undertaken on the instructions of George Frederick Insole (James Harvey's son, usually known as Fred),[6] to extend and modernise the building.

Rooms were enlarged and panelled, a new guest wing erected, central heating and electrical distribution inserted, and a large carriage porch was added.

[6] After World War I, in the light of Winston Churchill's decision to change the firing of the Royal Navy to oil and the loss of many European markets, the South Wales coal industry began to decline.

The family's fortunes suffered a similar downturn,[7] with the deaths of Fred, in 1917, and his son Claud, in 1918 whilst on war service in France.

[7][6] During World War II Insole Court was the regional Air Raid Warden HQ, and housed the Royal Observer Corps and Auxiliary Fire Service for Cardiff district.