Access to both Swan and Sand Islands is restricted to members of the golf and yacht clubs, Defence Department personnel, or by permit, with the only road guarded by a security gate.

The island was named by Acting Lieutenant John Murray when he entered Port Phillip in the ship Lady Nelson in February 1802.

[3] Swan Island has a long history of military use, being part of the fortifications built to protect the entrance to Port Phillip Bay from a feared Russian invasion during the Crimean War in the 1850s.

The Colonial Service torpedo boat HMVS Countess of Hopetoun's hull was scuttled here in 1925 after being stripped of valuable objects.

[4] On 9 April 2007, three Special Air Service Regiment soldiers from Perth completing a course on the SIAD were involved in a car accident after returning from a Queenscliff hotel and died.

A Royal Commission recommendation implemented abolished the Ministerial Directive permitting ASIS to undertake covert action.

In April 2004, with the passing of the Intelligence Services Act 2001, ASIS was permitted to be involved in paramilitary activities outside of Australia, including the passive use of weapons and self-defence techniques.

Anti-war protestors regularly protest at the training area, enter and are arrested by Victoria Police charged with Commonwealth trespass and plead guilty to the offence at court.