Fermor was born in 1702 in Pskov, Russia, of Scottish and Lutheran Baltic German descent.

He joined the Russian army in 1720 and distinguished himself at the Siege of Danzig during the War of the Polish Succession.

He preferred to associate mostly with other Germans in the service of Russia, something which caused resentment among Russian-speaking officers.

In 1757 Fermor led a Russian force that captured Memel in East Prussia[2] and was present at the Battle of Gross-Jägersdorf.

In 1760 he was in overall command of Russian forces during the Raid on Berlin, which saw them and their Austrian allies briefly occupy the Prussian capital before withdrawing.