He became royal governor of colonial South Carolina in 1755, serving until 5 April 1760, during the period of the French and Indian War.

His insistence on respecting the treaty rights of native peoples aggravated settlers on the frontier of South Carolina, who were encroaching on their territories.

In 1760, Lyttelton was appointed Governor of Jamaica, but he was recalled to England after he lost a standoff with the Jamaican House of the Assembly, and its leader, Nicholas Bourke, over who should stand costs for the island's defence.

However, the estates in Upper Arley passed to the late lord's sister Lucy, wife of Arthur Annesley, 1st Earl of Mountnorris.

In 1794, Lord Westcote was also created Baron Lyttelton in the Peerage of Great Britain.