It was substantially altered and remodelled in the 17th and 18th centuries by noted architects, including Sir John Soane.

In the 18th and 19th centuries, the Eliot family invested substantially in the estate, building numerous farmhouses, fishermen's cottages and other dwellings across the land.

In 1980 a small festival which had outgrown its site at Polgooth in mid-Cornwall approached the Port Eliot estate and asked if it could be held in the idyllic grounds.

The festival ran from 1981 to 1986, beginning with some 1,500 visitors over four days, and featured a mix of music, theatre and visual arts.

The burning down of the oldest tree in the park, looting of the village surgery and the robbing of stall-holders in 1985 prompted Lord Eliot and fellow organisers to make the 1986 festival the last.