Benson and the subsequent owners extended the original blockhouse to form a country house, landscaping the surrounding island to create ornamental gardens and lakes.

The 19th century saw continued building work by the castle's occupants, including the entrepreneur Colonel William Waugh, who erected various Jacobethan-styled extensions.

The wealthy stockbroker Charles Van Raalte led a lavish lifestyle at Brownsea at the start of the 20th century, using it to house his collection of antique musical instruments.

Brownsea Castle was built as a consequence of international tensions between England, France and the Holy Roman Empire in the final years of the reign of King Henry VIII.

Traditionally the Crown had left coastal defences to the local lords and communities, only taking a small role in building and maintaining fortifications, and while France and the Empire remained in conflict with one another, maritime raids were common but an actual invasion of England seemed unlikely.

[1] Modest defences, based around simple blockhouses and towers, existed in the south-west and along the Sussex coast, with a few more impressive works in the north of England, but in general the fortifications were very limited in scale.

[12][a] The castle needed regular investment: 101 piles were driven in during 1551, probably to combat coastal erosion, and in 1561 the town petitioned the Crown for help with further repairs and the provision of new cannons.

[16] In 1589, the Bountiful Gift refused to pull in for inspection, arguing that it had had a valid pass to leave, and Brownsea Castle fired on the vessel, killing two of the crew.

[22] Clayton probably did not live in the castle, however, and it fell into disrepair; by the end of the century the town of Poole refused to garrison the decaying defences.

[26] Benson argued that the building had not originally been a castle but rather a lodging house; it had not been built by Henry and therefore the Crown had no particular rights over it - rather, the previous owners of the island had simply allowed the town and the government to place artillery there.

[28] Humphrey extended the castle around its 16th century core to form a Palladian styled, four-storey tower with battlements, with new wings stretching away on three sides.

[14] He also built a walled courtyard with hot houses beside the castle, and landscaped the island with two lakes and a large number of fir trees at a cost of £50,000.

[32] A retired Indian Army officer, Colonel William Waugh, bought the island in 1852, hoping to turn it into a profitable pottery works.

[33] After a period of ownership by a Mr Faulkner, who continued the pottery scheme, the property was eventually resold in 1873 to the politician and lawyer George Cavendish-Bentinck for £30,000, who furnished the castle with an extensive collection of Italian Renaissance sculpture.

[43] On Florence's death in 1927 the castle was sold to Sir Arthur Wheeler, who promptly decided to auction off the remaining contents with the intent of demolishing the property.

The National Trust could only raise £25,000, so the remaining £75,000 was raised equally by the Boy Scout & Girl Guide Movements, by the forerunner of the Dorset WildLife Trust, who took over management of the North half the island, and by the John Lewis Partnership, who bought a 99-year repairing lease of the Castle and gardens for use as a corporate hotel by their employees, they gradually restored the buildings in stages and they remain the current tenants.