The Dixie family fortune was lost in the 19th century, and the house and estate were sold in the 1880s to pay gambling debts.

In the eighteenth century Sir Wolstan Dixie, the 4th baronet, had a reputation for being a pugnacious bully, with a penchant for using his fists to settle any dispute, which often set him at odds with his neighbours and even ex-employees.

In March 1732 he appointed the young and impoverished Samuel Johnson (1709–1784) to a position of usher at the school, even though he did not have the required university degree.

[7] In 1875, he married Florence Douglas (1855–1905), who in her lifetime was well known as a writer, feminist, big game hunter, war correspondent, and suffragette.

Sir Alexander and Lady Florence left Bosworth in the early 1880s and went to live at Glen Stuart House on Lord Queensberry's Kinmount estate in Dumfriesshire, Scotland.

The Bosworth estate was purchased in 1885 by Charles Tollemache Scott, who made numerous improvements to the building and added his initials to some of the iron guttering, which can still be seen to this day.

Among other changes Tollemache Scott made, the cellar gates were replaced with cell doors from the Newgate Prison in London.