Albany was built in 1771–1776 by Sir William Chambers for the newly created 1st Viscount Melbourne who had bought the land and residence (Piccadilly House) it was to replace from Henry Fox, 1st Baron Holland for £16,500.

Residents have included the poet Lord Byron, the future prime minister William Ewart Gladstone and numerous members of the aristocracy.

[5] The Albany Trust is named after the building, as it held its inaugural meetings there in the late 1950s, at the home of its founding trustees Jacquetta Hawkes and J.

Still earlier is the hero of Benjamin Disraeli’s novel Sybil (1845), Charles Egremont, who lives there; he has a portrait by Cristofano Allori hung over his fireplace halfway through the book.

In Oscar Wilde's play The Importance of Being Earnest (1895), the character John (Jack) Worthing has a set at Albany (number B.4), where he lives while staying in London under the assumed name of Ernest.

In G. K. Chesterton's Father Brown Stories, in "The Queer Feet" (1910), the character Mr Audrey "[looks] like a mild, self-indulgent bachelor, with rooms in the Albany -- which he was".

In the comic short story "Uncle Fred Flits By" (1935) by P. G. Wodehouse, the young gentleman Pongo Twistleton resides in Albany.

In the film Kind Hearts and Coronets (1949), Louis Mazzini takes a small set at Albany as he moves up the social ladder.

In the James Bond novel Moonraker by Ian Fleming (1955), Max Meyer, the bridge partner of Sir Hugo Drax, was said to live in Albany.

Simon Raven's Alms for Oblivion novels (including 1974's Bring Forth the Body) feature Somerset Lloyd-James, a politician and resident of Albany.