Throughout the medieval period, the estate on which Abberley Hall stands was a possession of the de Toeni family of Elmley Castle, Worcestershire.
A 'chief messuage' was mentioned in 1309, and Habington says that "Abberley seemethe to have byn of owld the principall seat of the Lord Thony in this shyre".
When Bromley died in the 1830s, leaving seven daughters and no son, his executors decided to sell the estate, which comprised less than 800 acres (3.2 km2) and was heavily encumbered by debts.
The house was extended and redecorated internally by J P St Aubyn for a later owner, John Joseph Jones, an Oldham cotton magnate, in about 1883.
St Aubyn also built a clock tower (now Grade II* listed)[5] in the grounds in 1883–85, and the main entrance lodge.