Originally established as a British naval depot, the island developed slowly from the early colonial days, with the greatest spurt occurring during World War II.

This period saw massive building programs including the construction of the Captain Cook Graving Dock and land reclamation by 1945 that joined the island to the mainland at Potts Point.

[2] Within 2–3 years of the arrival of the British in 1788, the Cadigal people's patterns of life were increasingly difficult to sustain, due to conflict, loss of resources, disease and the attractions of developing colonial towns.

[5] Others have questioned this rapid demise when, for example, in 1795 it was reported that a large number of people had gathered in Farm Cove, 500 metres (1,600 ft) west of the precinct, for an initiation ceremony.

Even more tangible evidence of the invasion of the Cadigal domain was provided by some of the early gardeners when three men carved their initials into a rock on the island's northern hill: "FM", "IR" and "WB" with the year "1788".

[2][3]: 41–56 In 1789, the Row Boat Guard was established to patrol the harbour for smugglers, and a watch house was built on the island, but later transferred to Watsons Bay because of a lack of fresh water.

[2] The expansion of the wool industry and the gold rushes showed Australia was more than a stopping point between the Pacific and Indian oceans, and the use of Sydney as a base during the Maori Wars in New Zealand emphasised its strategic role.

At the same time the agitation for, and then the granting of, responsible government to NSW in 1856 resulted in the elected colonial politicians campaigning for the use of Sydney Harbour as a base for the Royal Navy in the South Pacific.

[2][3]: 80–97 The navy's then use of Fort Macquarie as its main residence was not secure due to that site's co-use as part of the Sydney Harbour military defence system.

The Navy took control of the island in 1858 at a time that coincided with the development of the separate British fleet, the "Australia Station" – to guard against aggression by external naval forces.

The building comprised a four-storey (semi-basement and three upper floors) late Victorian Warehouse of load-bearing polychrome bricks with sandstone string courses, cornices, sills, copings and granite thresholds to doors.

That year saw the entire Garden Island depot completed and handed over to Rear Admiral Sir Cyprian Bridge, Commander-in-Chief, on behalf of the Admiralty.

1900 saw Australia's participation in the Boxer Rebellion, particularly the New South Wales Naval Brigade and the conversion to troopship at Garden Island of the steamship SS Salamis.

[3]: 157  The ownership question of Garden Island was raised again at the end of the decade with Britain's intention to withdraw vessels on the Australia Station due to the Australian Government's moves to determine its own naval policy.

[2][3]: 165 Garden Island was formally handed over from the British Admiralty to the Commonwealth Government in 1913 and coincided with the establishment of the first wholly owned and operated Royal Australian Navy.

On 12 October 1923 the State Government formally revoked the earlier dedications and by 1924 issued an ultimatum to the Commonwealth for possession of the island, or a declaration of that entitlement, plus claims for profits owed for its use.

[2] Late 1930s world political events led to a re-assessment of the need for Australia's national naval defence and the importance of facilities such as Garden Island.

As Tom Frame notes, construction of the graving dock and ancillary equipment such as cranes, "would be the greatest engineering feat in Australia's history, surpassing even the Sydney Harbour Bridge.

[2][3]: 191 World War II saw a reinstatement of the critical value of Garden Island to national defence and the facility immediately undertook extensive vessel repairs and modification work.

The principle targets were the heavy ships in port at that time, including the cruiser USS Chicago tied up at No.2 Buoy on the eastern side of Garden Island.

Officially opened on 24 March 1945, the dock was named in honour of Captain James Cook RN, with the celebratory ribbon cut by the bow of the frigate HMAS Lachlan.

[2][3]: 209–210 On part of the reclaimed land there is a 115.8-metre long (380 ft) dock and a complex of buildings on the western side of Garden Island and on the eastern edge of Woolloomooloo Bay.

The island had hummocks at its north (24metres high) and south (20 metres) ends, with a lower flat between them until substantial re-forming began in 1885 as part of the development of the naval base.

[2][14][15] Separately-listed on the Commonwealth Heritage List,[16] the Rigging Shed and Naval Chapel is a two-storey Victorian Italianate stuccoed brick building having stone sills, arches and columns, with original timber beams replaced by steel.

The Chapel incorporates a unique stained glass commemorative window to Australia's first two submarines, AE1 and AE2, based at the Island and lost at Rabaul (1914) and near Gallipoli (1915) respectively.

The main machinery house is situated on top of the boom, making the total height of the complete structure 61.9 metres (203 ft) from wharf level.

The precinct has important associations with historically significant people, notably king Bungaree, Governor Lachlan Macquarie, Commodore Loring, Rear Admiral Tyron, and Colonial Architect James Barnet.

The precinct is rare as the only example of a fleet base in New South Wales, and illustrates the combination of industrial, military and recreational uses that have characterised the shores and islands of Sydney Harbour since 1788.

Garden Island has long been associated with maritime activities such as victualling and repairing naval vessels, and with the building and operation of harbourside defences.

The precinct, although subject to disturbance especially during the building of the graving dock, retains potential for maritime archaeological remains associated with the early wharf and jetty facilities, and depositional material from the shore and fleet units.