[1][2] Elizabeth Bay had been the site of a fishing village established by Governor Macquarie (1810–21) in c. 1815 for a composite group of Cadigal people – the indigenous inhabitants of the area surrounding Sydney Harbour – under the leadership of Bungaree (d. 1830).

He was elected a fellow of the Linnean Society of London in 1794 commemorating the great Swedish naturalist Carl Linnaeus, whose Species Plantarum (1753) became the internationally accepted starting point for all botanical nomenclature and served as its secretary from 1798 to 1825.

Botanist Robert Brown, Macleay's close friend and suitor of his eldest daughter Fanny, a competent botanical artist, named the plant genus Macleaya in his honour.

[1][2] In enforced retirement from 1817 when his department was abolished at the end of the Napoleonic Wars, Macleay's finances were stretched to support a large family (10 of 17 children survived to adulthood), town and country residences, and his obsessive collecting of insects.

A spacious garden, filled with almost every variety of vegetable; a trellised vinery; a flower garden, rich in botanical curiosities, refreshed with ponds of pure water and overlooked by fanciful grottoes; a maze of gravel walks winding around the rugged hills in every direction, and affording sometimes an umbrageous solitude, sometimes a sylvan coup d'oeil, and sometimes a bold view of the spreading bays and distant headlands – these are living proofs that its honourable proprietor well deserved the boon, and has well repaid it.

Although Nurseryman Thomas Shepherd had practised as a landscape gardener many years previously in England and his 1835 (public) lecture (in Sydney) included suggestions for the further improvement of the Elizabeth Bay estate, he does not claim credit for involvement, however informal, in its design.

Nurseryman Thomas Shepherd wished others to emulate this:[1] The high lands and slopes of this property are composed of rocks, richly ornamented with beautiful indigenous trees and shrubs.

He has also obtained the benefit of a standing plantation which it might otherwise have taken twenty or thirty years to bring to maturity.The bush was planted with specimen orchids and ferns to enhance its botanical interest, which could be enjoyed in the course of a "wood walk".

It is possible that Macleay's son William Sharp, after his examination of his father's finances upon joining the family in Sydney in 1839, called for a halt to the building of the house.

Six years later in 1845 in the midst of a colonial financial crisis, he sold them to the newly completed Government House, Sydney, where three of the original rosewood veneer tables still have pride of place.

William Sharp Macleay (1792-1865), public servant, scholar and naturalist, and eldest son, inherited his father's insect collection, and stayed alone at the house until his death in 1865.

George progressively subdivided the estate and sold leaseholds of a substantial portion and leased the house to his cousin William John Macleay and his wife Susan.

[1] From 1948 to 1950 Sydney City Council using landscape architect Ilmar Berzins created the Arthur MacElhone Reserve on what had been Elizabeth Bay House's famous lawn (three lots, unsold in the 1927 subdivision).

Berzins' design incorporated Macleay's curving retaining wall and protruding sandstone ledges or benches and a grotto facing onto the footpath on Billyard Avenue below.

[19] In November 2020, Hayley Mary from The Jezabels released the video to her single The Chain, which was directed by Tyson Perkins and filmed at Elizabeth Bay House.

The entablatures and fluted pilasters of the doorways, the tapering Grecian architraves and panelled reveal shutters of the windows and the plaster cornice and frieze decorated with laurel wreaths.

In 1845 Macleay sold these to the newly completed Government House, Sydney where three of the original drawing room furnishings, being rosewood veneer tables, still have pride of place.

The pavilion was intended for the extremity of nearby Macleay Point, facing Rushcutters Bay and which was poetically named Cape Sunium after the peninsula east of Athens with its picturesque ruined temple.

Elizabeth Bay House's incomplete state reflects the 1840s depression which devastated a class of prominent colonial civil servants, pastoralists and merchants.

This is indicated by proposals to refurbish the house as a museum for the 1938 sesquicentenary of white settlement, Professor Leslie Wilkinson's ownership share in Elizabeth Bay Estates Limited (1926–1935), the acquisition of the property by the Cumberland County Council in 1963 for its historic significance and the 1972–76 restoration by Clive Lucas, one of the first modern, scholarly conservations in Australia.

The house's association with Alexander Macleay reflects the exploration and settlement of Australia concurrently with the extraordinary growth in scholarly interest in the natural sciences at the end of the 18th century.

Elizabeth Bay House and its garden were visited by prominent Australian, British, American and European scientists and intellectuals throughout most of the 19th century, being a focus for Australia's role in an international scientific community.

[1] The estate is the location of the Aboriginal settlement of Elizabeth Town, established by Governor Macquarie and under the leadership of Bungaree and is a significant Aboriginal-European contact site.

The quality of the Greek Revival styling of the house marks a transition in John Verge's career from relatively restrained commissions such as Lyndhurst, Glebe, to more sophisticated designs such as Camden Park.

Elizabeth Bay House's relationship with the villas Tusculum and Rockwall, Potts Point provides a rare key precedent for town planning in Australia.

Elizabeth Bay House has had a significant, albeit limited, impact on the architecture of NSW and the appreciation of colonial buildings through the developing conservation movement.

Elizabeth Bay House is a rare superlative example of a representative class of Greek Revival villas constructed by New South Wales' most fashionable architect, John Verge, for prominent colonial civil servants, pastoralists and merchants.

[1] As at 22 September 2003, the grotto and associated stairs, balustrade and retaining walls are ornamental structures created between 1832 and 1835 to embellish the then 22-hectare (55-acre) garden of Elizabeth Bay House, built between 1835 and 1839 by Alexander Macleay (1767–1848), Colonial Secretary of New South Wales (1826–1837).

[2] The siting of Elizabeth Bay House and the layout of its drives, garden terraces and grottoes was carefully planned to maximise vistas and the dramatic Sydney Harbour topography.

These included the demolished spired stables (c.1828, designer unknown), a gardener's cottage (1827), rustic bridge and pond (c.1832) and the extant grottoes, retaining walls and stairs.