The first mentally ill patients in Western Australia were cared for in temporary accommodation, including the wreck of Marquis of Anglesea, the Round House and the Colonial Hospital.

Official care began with the transfer of ten convicts from Perth Gaol to a new asylum located in Scott's Warehouse (corner of Croke and Cliff streets) in November 1857.

[9]: 1  The design, in colonial gothic style was by Lieutenant-Colonel Edmund Henderson, the Comptroller-General of Convicts in Western Australia, and built to accommodate 50 people.

During World War II it became the headquarters for the American armed services based in Western Australia, who built the asbestos-clad laundry building on the north-east corner of the site.

After many years of lobbying for state and federal government funding, a major restoration project commenced in 1970 and in 1972 it housed the Western Australian Maritime Museum (since relocated to Victoria Quay), and Fremantle Arts Centre.

In January 2007, conservation works were completed with the gable finials on the west façade restored to their original state, following their demolition at the turn of the 20th century.