Wellesley, now Lord Wellington, was not satisfied that Spencer was up to the responsibilities of second-in-command and he was replaced by Thomas Graham.

Promoted to brigadier general, he participated in the unsuccessful 1797 Santo Domingo campaign against Toussaint Louverture.

[5] In 1801, Spencer served with General Sir Ralph Abercromby's army in Egypt at the Battle of Alexandria.

The Dos de Mayo Uprising of Spain against Napoleon found Spencer in command at Gibraltar.

At Mondego Bay he joined his force to the 8,123 troops that General Arthur Wellesley brought from Cork in Ireland.

[7] Later that year, he was replaced by Thomas Graham, 1st Baron Lynedoch because of pessimistic letters which he had sent back to England.

Spencer met Princess Augusta Sophia, the second daughter of King George III and Charlotte of Mecklenburg-Strelitz, about 1800, and a romantic friendship had developed by about 1803.

According to Augusta, the couple conducted their relationship with great discretion as they were not of equal rank; news of such an attachment was feared to disturb the King's precarious mental health.