It adopted its name "Milson's Point" in the 1830s to refer to the area near the northern side of Sydney Harbour.

In the early 1820s,[5] Milson settled in the vicinity of Jeffrey Street, Kirribilli, on 120 acres of land he leased from Robert Campbell.

In 1826 a bushfire raged through the area destroying Milson's home, orchard and dairy and farm[8] which he subsequently rebuilt (refer to 1840s map).

[6] Milson lived in the vicinity of Jeffrey Street until 1831 when he built a new home, "Brisbane House", on his 50 acres facing Lavender Bay.

[12] The last of the family's holdings in the lower North Shore area were resumed in the early 1920s for the construction of the Harbour Bridge and associated roadways.

The harbour-side amusement park features a collection of preserved fantasy architecture in the style of Art Deco.

It was closed by the state government in 1990, despite earlier promises by Premier Nick Greiner that it would remain open and a legal challenge by the school's parents.