[3] The estate, in near ruin, was purchased in 1666 by Henry Bennet, Earl of Arlington and Secretary of State to the newly restored King, Charles II.

He constructed a grand house in the French style, built around a central court with large pavilions on each corner.

In 1672 Charles II arranged a marriage between nine-year-old Henry FitzRoy, his illegitimate son by Barbara Villiers, and Isabella Bennet, the Earl of Arlington's five-year-old heiress.

In about 1750 their son, the second Duke, decided to re-model the house and employed Matthew Brettingham,[5] who supervised the execution of William Kent's and Lord Burlington's design of Holkham Hall in Norfolk.

According to the Sporting Magazine, February 1793: "About the year 1735 he [the 2nd Duke of Grafton] kept foxhounds at Croydon and went to London very early on the days he hunted.

The old Duke used to complain bitterly of the interruption he met with (in crossing the Thames at Westminster) for the delay and inattention of the ferry man, etc., by which he often lost several hours of a fine morning before he arrived in Croydon.

The old park was designed by the diarist John Evelyn,[2] a noted landscape gardener and an expert on trees, with a canal, straight rides and avenues.