The house was built between 1746 and 1750 for the wealthy merchant and philanthropist Paul Fisher, by Isaac Ware, a nationally renowned architect and translator of Palladio's works.

There is a World War II Air raid shelter accessible beneath the steps which is visible from the South Façade.

Original rococo plasterwork, by Joseph Thomas, survives in a number of interior rooms.

The vantage point of the house offered a view of the Avon, of the city of Bristol, and of the Bath hills.

In 1907 May Staveley a lecturer in the History Department of the University of Bristol, started lobbying for need to provide a hall for her students.