He was notably a sixth-generation descendant of King Charles II and the 1st Duchess of Cleveland; the surname FitzRoy stems from this illegitimacy.

Charles' half brother Robert FitzRoy would become a pioneering meteorologist and surveyor, Captain of HMS Beagle, and later Governor of New Zealand.

Charles FitzRoy was educated at Harrow School in London, before receiving a commission in the Royal Horse Guards regiment of the British Army at the age of 16.

Sir Charles was appointed as the eleventh Governor of Prince Edward Island off the coast of Canada on 31 March 1837, and was granted a knighthood just before his departure.

Soon after his arrival he was asked to use his influence to procure the disallowance of an act of the Tasmanian legislature imposing a duty of 15% on products imported from New South Wales.

Fitzroy brought before the British government the advisability of some superior functionary being appointed, to whom all measures passed by local legislatures should be referred before being assented to.

In the long discussion over the separation of the Port Phillip district, Fitzroy showed tact and himself favoured bi-cameral legislatures for the new constitutions.

Transportation of convicts ceased, the Sydney University was founded, a branch of the royal mint was established and responsible government was granted.

The Governor stayed at the Surveyor General's Inn, operated by former boxing champion Edward "Ned" Chalker (sometimes Charker).