Cap Diamant's strategic value was identified by Samuel de Champlain in 1608 and led him to found Quebec City at the base of the escarpment.

[8] After the fall of Louisbourg in 1745, considerable work on the battlements took place under the direction of military engineer Gaspard-Joseph Chaussegros de Léry.

[10] That opinion finally shifted following the War of 1812;[11] as part of a wider improvement of Canada's defences coordinated by the Duke of Richmond, then Governor-in-Chief of British North America,[11] the existing star fort was built between 1820 and 1850 under the direction of Lieutenant-Colonel Elias Walker Durnford of the Royal Engineers.

Intended to secure Quebec City against the Americans and serve as a refuge for the British garrison in the event of attack or rebellion, the Citadelle incorporated a section of the French enceinte of 1745 and the layout was based on Sébastien Le Prestre de Vauban's design for an arms, munitions, and supplies depot, as well as a barracks.

[12] From the late 19th century, living conditions for soldiers at the fort gradually improved; canteens were opened and the casemates were made more comfortable.

[12] The preservation of much of the fortifications and defences of Quebec is due to the intervention of Governor General of Canada the Marquess of Dufferin and Ava, who also established the Citadelle as a viceregal residence in 1872,[13] reviving a tradition dating to the founding of New France.

The Quebec Conferences of 1943 and 1944, in which Governor General of Canada the Earl of Athlone, Prime Minister of Canada William Lyon Mackenzie King, British Prime Minister Winston Churchill, and US President Franklin D. Roosevelt discussed strategy for World War II, were held at the Citadelle of Quebec.

[15] The Citadelle is a functioning military installation for the Canadian Armed Forces, as well as an official residence of both Canada's monarch[16] and its governor general.

As is done at the other federal royal residence, Rideau Hall in Ottawa, Canadian award presentations and investitures and ceremonies for both incoming and outgoing ambassadors and high commissioners to Canada are held at the Citadelle.

[24] The fort is an uneven star shaped citadel and comprises four bastions and three straight curtain walls, all constructed with locally quarried sandstone.

The residence today has a total of 153 rooms over 4,459 square metres (48,000 sq ft),[25] including offices for the governor general's secretary.

[26] The entrance into the original area of the residence is through a set of double doors beneath a neo-classical porch bearing the words GOUVERNEUR GÉNÉRAL on the frieze and the crest of the Royal Arms of Canada in the tympanum.

Within is a foyer with marble tile floor, a stair descending to the basement, and, through another set of doors in a screen with translucent leaded glass sidelights and fanlight, is a hall; both rooms are in the Georgian style, in beige, cream, and gold.

This new wing was built at the east end of and at a slight angle to the 1831 structure, the roof being copper and the exterior walls of the same masonry as the adjacent buildings, but, using more regulated block sizes and a flatter relief of pilasters and windows, as well as less detailing overall.

The wing contains a separate entrance and ceremonial foyer with twin spiral staircases ascending to a piano nobile; the stairs have wood handrails with aluminum pickets and between, the flights, is a niche for sculpture.

The former two areas are fully barrel vaulted and linked together by a continuous, narrow skylight, under which crystal pendants of different lengths hang and transfer the natural light into the rooms.

[13] The residence is furnished with pieces from the Crown Collection, mostly in New France style, antique furniture mixed with contemporary Canadian art.

The west-facing facade features restrained decorative detailing of pilasters supporting a flat cornice along the length of the five-bay façade.

It is a long, rectangular, two story masonry building built of smooth limestone with a hipped roof clad in sheet copper.

Building 5, the Former Powder Magazine, is a low, rectangular stone structure with a gabled roof and a surrounding protective blast wall.

[32] Located within the King's Bastion and adjacent to the governor general's residence, Building 10 is the former military prison and presently the museum annex.