Count Alexander Ivanovich Osterman-Tolstoy was the son of the Lieutenant-General Ivan Matveevich Tolstoy (1746–1808) and his wife Agrafena Ilyinichna, nee Bibikova, "of ancient Tatar stock", and which, as per another member of the family, historian Nikolai Tolstoy, "may account for Alexander's dark complexion.

In 1796 his two childless great-uncles, Fedor and Ivan Osterman, brothers of his paternal grandmother, gave him their family name, the title of count and a large fortune.

Count Osterman-Tolstoy remained in service until 1826, after Alexander's I death he retired and started traveling around Europe and the Middle East in the company of the scholar Jakob Philipp Fallmerayer.

Fearlessness, courage, and endurance in battle were his characteristics as a military officer.” Count Osterman-Tolstoy was married to Princess Elisabeth Alexeevna Galitzine (1779–1853) from 1799; they had no children.

There is an engraving, published in Pisa in 1827, on which Count Osterman-Tolstoy is depicted sitting beside a pram with a sleeping baby and two older children playing nearby; the inscription says, ‘Je me flatte que c’est les derniers faries (sic).

A 55 ans il est temps de faire la clôture.’ (I flatter myself with the thought that it’s my last extravagancy.