The origins of the present house are a mansion rebuilt and extended by William Baker for the 1st Earl of Powis in the mid-18th century.

Elected Member of Parliament for Ludlow in 1818,[a] the following year Clive commissioned his friend Charles Robert Cockerell to remodel the house.

Cockerell's subsequent work on the excavations at Bassae in Greece provided inspiration for some of the decoration at Oakly Park.

The vestibule is a circular room dating from the younger Haycock's work, although Cockerell added a "saucer dome".

[1] The conservatory was the last element of the first phase of building, the breakfast and dining rooms dating from the second, when Clive, by then Windsor-Clive following his marriage to Harriet Hickman, joint heiress to the Earl of Plymouth, required more space for his expanding family.