The castle was damaged in the English Civil War and then restored in the 18th and early 19th centuries by Charles Howard, 11th Duke of Norfolk.

Roger de Montgomery was declared the first Earl of Arundel as the King granted him the property as part of a much larger package of hundreds of manors.

[5] The castle then passed to Adeliza of Louvain (who had previously been married to Henry I) and her husband William d'Aubigny.

[4] The folly that still stands on the hill above Swanbourne Lake was commissioned by and built for the Duke by Francis Hiorne at this time.

The work, which was done to the designs of Charles Alban Buckler and undertaken by Rattee and Kett of Cambridge, was completed in the late 19th century.

He created an independent charitable trust to guarantee the castle's future, and oversaw restorative works.

The gardeners and volunteers "have worked wonders with their bold and innovative plantings", according to an April 2020 report by Country Life.

The garden's centerpiece, Oberon's Palace, is a stunning pavilion that features a shellwork grotto and a fountain that supports a golden corona when the water spurts.

[16] The cricket field in the castle grounds has, since 1895, seen matches involving teams from local youths to international sides.