On the south the boundary is formed by Exeter City Central Library and on the south-east by the Phoenix Arts Centre and the Royal Albert Memorial Museum.
[2] The site was originally part of William the Conqueror's defences for Rougemont Castle during the Norman Conquest of England.
When William ordered the construction of Rougemont Castle in 1068 a defensive ringwork was included which characterises the shape of gardens today.
It has been noted that the ditch in the gardens seems to be much larger than the apparent need for defence; this could be because it may have been the site of the Roman quarry which provided the stone from which the original city walls were built.
[6] Until 2009, they were used by the Northcott Theatre to stage Shakespearean plays in the summer—the city wall and defensive ditch forming a natural backdrop and amphitheatre for the performances.