The large Victorian Regency and Italianate sandstone manor, completed in stages based on designs by James Barnet and Walter Liberty Vernon, occupies the tip of Kirribilli Point.

[3][4] Before the arrival of British settlers in Sydney Harbour, the Aboriginal Cammeraygal people lived along the Kirribilli and Milsons Point foreshores, and in the surrounding bushland.

It is likely that the farm was located at the Jeffrey Street end of Kirribilli (not near Admiralty House) and was named "Huntershill" by Thomas Muir, after his father's home in Scotland.

Four years later the Colonial Secretary recorded that the land grant to Lightfoot was cancelled and given to Robert Ryan in 1800 with no mention of the intermediate (private) sale to Muir.

[14] In 1842, the 2.0-hectare (5-acre) site where Admiralty House now sits was leased to the Collector of Customs for the Colony, Lieutenant-Colonel (later full Colonel) Gibbes, MLC.

On the superb Kirribilli Point location, Gibbes erected, between 1842 and 1843, a graceful single-storey house with wide verandahs and elegant French doors.

Gibbes engaged James Hume, a well-known builder who dabbled in ecclesiastical architecture, to supervise the construction of the building and its stables.

Once completed, Gibbes' L-shaped residence featured a plain, yet stylish, double façade to maximise the building's magnificent, sweeping views across Sydney Harbour.

These views enabled Gibbes to monitor shipping traffic in and out of Darling Harbour and, more importantly, Circular Quay, where the Sydney Customs House was situated.

Today, Wotonga forms the core of Admiralty House and the building's 180-degree, east–west panoramic sight-lines are even more spectacular than they were in Gibbes' day, owing to the subsequent high-rise growth of Sydney's CBD.

This reputed connection to the British monarchy adds spice[citation needed] to the house on Kirribilli Point's subsequent role as a vice-regal establishment.

[15] In 1849, Robert Campbell died and the executors of the estate sold the property, comprising the house and 2.0 hectares (5 acres) land, to Gibbes for about A£1,400.

On 27 December 1851, Gibbes, who was contemplating a departure from the Customs Service at the age of 64, sold the property to James Lindsay Travers, a merchant of Macquarie Place, Sydney, for £1,533.

(Gibbes subsequently changed his mind about leaving his position as head of the NSW Customs Department; instead, he leased Greycliffe House at Shark Beach, Vaucluse, from the Wentworth family and remained in Sydney for the better part of eight years, eventually retiring to Yarralumla homestead, now the official Canberra residence of the Governor-General of Australia, in 1859.)

In 1856, Lieutenant-Colonel George Barney, a Royal Engineers officer, lived in "Wotonga" and designed and supervised the installation of a battery of five, 8-inch (20 cm) muzzle-loading guns on Kirribilli Point, as well as constructing the martello tower at Fort Denison.

In 1866, it was let to Frederick Lassetter and subsequently to James Robert Wilshire, a former Lord Mayor of Sydney and a Member of the New South Wales Legislative Council from 1858 to 1861.

In April 1874, Wotonga House was auctioned and bought for £10,100 by Thomas Cadell, a Sydney merchant and member of the New South Wales Legislative Council from 1881 to 1896.

In 1912, the Government of New South Wales decided to put the building to public purposes once more, leaving the Governor-General of the period, Lord Denman, without a Sydney residence.