[1] His father was a mayor of Southampton, a Justice of the Peace, and also a strict Calvinist who had inherited a successful wine business from his childless uncle George Robinson (1700–1776).

His marriage to Celia Anne Crowcher (1779–1848) on 24 November 1798 at St Mary's Church, Portsea was a failure and was annulled in 1802.

He led a reclusive life and, in 1825, left Southampton never to return, spending his time between Calais in France and Newington, Surrey, near London.

[9][10] He wished his house to become a museum designed to promote the study of natural history, astronomy, antiquities, classics, and oriental literature.

Using the remaining funds, the City of Southampton founded the Hartley Institute (1862) by combining a museum, library and venue for public lectures.