[3] In the Dharug language, Goat Island is also known as Memel or Me-Mel, meaning the eye, by the indigenous Eora people of Port Jackson.

[3][8] By the early 1830s increasing amounts of gunpowder for public works were in storage in Sydney prompting Governor Bourke to implement Darling's proposal for the construction of an arsenal or magazine on Goat Island.

In 1833 gangs of convicts started work quarrying stone and levelling ground on a site at the south-western side of the island.

In 1834, three portable wooden houses surrounded by a stockade were erected on the island to avoid the loss of time involved in the daily ferrying of the convicts from the hulk.

Newly arrived Commanding Royal Engineer George Barney took control of the construction in January 1836 and immediately instructed Thomas Bird to prepare plans of the buildings already underway.

During construction, the magazine, cooperage, barrack and convict accommodation, wharf, stone walls, well or tank, garden, water channels, wet ditch and blacksmith's shop had been erected.

This was finally agreed to in January 1837 when Barney proposed the use of convict labour to construct a wet ditch across a small segment of Goat Island.

Colonial Architect Mortimer William Lewis prepared a design for the proposed Water Police station which went to tender in June 1838.

[10][3] The Ordnance or Queen's Magazine on Goat Island was intended to store gunpowder belonging to the British government and by arrangement that owned by the Colonial administration.

[3] The matter was resolved in 1850 when Colonial Architect Edmund Blacket was requested to prepare a plan and estimate for a merchants' gunpowder magazine on Goat Island.

This was a building of much lighter construction, and was intended for the storage of civilian explosives belonging to the colonial government and local merchants.

[3] In 1861 the Colonial Secretary raised concerns that the quantity of gunpowder stored on Goat Island was sufficient to "send half of Sydney to the other world."

The island is believed to have been used for a period in that year as a bacteriology station, for the investigation of the major outbreak of bubonic plague in the nearby Rocks district, but firm evidence for this usage is lacking.

What is certain is that by 1901 the island had become the depot for the Sydney Harbour Trust, responsible for the maintenance of that body's significant fleet of tugs, dredgers and other floating plant.

The Trust used Goat Island as a depot constructing wharves, berthing facilities, coal-store, 4 cottages, Harbour Masters Residence and workshop as well as making major alterations and additions to the former barrack and cook house.

The Premier will establish a working group to investigate means of transferring management while continuing existing operations on the island, protecting its heritage and maintaining public access.

Former state MP Linda Burney's last question to Parliament before she left for federal politics was about returning ownership of the island.

Minister for Aboriginal Affairs Leslie Williams said "future opportunities (included) increasing public access and sharing cultural experiences".

Prior to this, $42 million is to be spent on the island's restoration, including repairs to seawalls and buildings as well as upgrades to the wharf, access and services, and the removal of asbestos.

On 19 November 2013 Kings of Leon performed on the island in front of 600 people presented by Foxtel's Channel V. As at 24 March 2000, the Colonial Magazine is historically significant as probably the oldest surviving magazine built to store merchant's powder in Australia, evidence of the growing need for storage of privately owned gunpowder in the expanding colony of New South Wales.

It has historical associations with its designers, Colonial Architects Edmund Blacket and Alexander Dawson, and despite the loss of original structural arrangement retains some ability to demonstrate its former use as a magazine facility for the storage of gunpowder.

The building is aesthetically significant mainly for the technical innovation of its design which departed from the military standard typified by the adjacent Queen's Magazine.

[20][3] Goat Island was listed on the New South Wales State Heritage Register on 2 April 1999 having satisfied the following criteria.

After 1901 it was the shipyard and base for port management operations by the Sydney Harbour Trust and its successor the Maritime Services Board.

[3] The place is important in demonstrating aesthetic characteristics and/or a high degree of creative or technical achievement in New South Wales.

The Queen's magazine's powerful architectural qualities are complemented by the unusual design of the contemporary stone cooperage, barrack buildings and perimeter walling.

The magazine's aesthetic qualities are further enhanced by the topographical setting on the south western edge of the island and by the curved alignment of the stone security wall.

The variety, extent and pattern of wharves, is unusual in such a concentration and provides a rich visual interplay between the rhythm of the piles and the rocky foreshore.

It is an example of the State's response to the need to provide a safe storage facility and distribution point for both publicly (both Imperial and Colonial) and privately owned explosives.

Surviving physical evidence can demonstrate the life styles and working conditions of a diverse range of occupants and staff on the island, during all phases of development from Pre-European to the late 20th-century activities of the Maritime Services Board.