From 1720 to 1770 Port-la-Joye, later named Fort Amherst, served as the seat of government and port of entry for settlers to the island while under both French and British control.

"[2] The first European settlers in the area were French military personnel from Fortress Louisbourg who founded a settlement in 1720 named Port La-Joye on the southwestern part of the harbour opposite the present-day city of Charlottetown.

The wood barracks were poor protection from harsh winters when wind, rain and snow swirled between picket walls and rotten planked roofs.

When the French commander of Louisbourg capitulated to the invasion force composed largely of New England irregulars, this also resulted in the de facto surrender of Île Saint-Jean (present-day Prince Edward Island).

[8] To regain control of Acadia for France, Jean-Baptiste Nicolas Roch de Ramezay was sent from Quebec to the region in 1746 to join forces with the Duc d'Anville Expedition.

[9] Upon arriving at Fort Beausejour on the Isthmus of Chignecto, he sent French officer Boishébert to Île Saint-Jean on a reconnaissance to assess the size of the British forces.

French military forces constructed a star-shaped fort on the site between 1748 and 1749 in a style influenced by Sébastien Le Prestre de Vauban.

In late August a small British fleet of four ships carrying 500 soldiers under command of Lord Rollo arrived at Port-la-Joye.

Lord Rollo, travelling aboard HMS Hind (1749), had been told to expect approximately 300-500 Acadians but was surprised to find roughly 3,000-5,000 instead.

After the British seized control of Port-la-Joye they replaced the rudimentary French fortification with a new stockade fort immediately to the east (toward the water).

A mutiny took place among the garrison at Fort Amherst in 1762, resulting in courts-martial at Louisbourg for the main people involved; demotions and hundreds of lashes by cat o'nine tails and one execution.

From there, Holland, his deputy surveyor Thomas Wright, engineers, volunteers, and soldiers from Fort Amherst set out to complete the survey, enduring harsh conditions through the winter.

On November 17, 1775, during the American Revolutionary War, the colonial capital of St. John's Island was attacked by Massachusetts-based privateers in the Raid on Charlottetown (1775).

In 1801 the British Army upgraded its defences of the colonial capital by establishing the Prince Edward Battery on the western edge of Charlottetown in present-day Victoria Park.

A variety of landowners farmed the property until 1959 when it was purchased by the federal government for preservation, which had designated it a National Historic Site the previous year.

A 2007 article, "The Un-Canadians", in The Beaver, includes Amherst on a list of those in the history of Canada who are considered contemptible because he "supported plans of distributing smallpox-infested blankets to First Nations people.

"[17] In February 2016, a spokesperson for Parks Canada said it would review the matter after a proper complaint is filed, and after consultation with the Historic Sites and Monuments Board.