He was the only son of the "fermier général" Pierre Grimaud du Fort (1692–1748) and his wife, Marie Antoinette Felicité de Caulaincourt (b.
This phase of the hotel – of which nothing remains today – constituted an important step in the evolution of the taste, and contributed to his entry into aristocratic circles of the Faubourg Saint-Germain.
Marie Louise died giving birth to him, and Pierre (a patron of the arts like his father) began travelling Europe for consolation, gathering famous collections of paintings and sculptures.
While traveling, he also met his second wife, Princess Maria Anna of Hohenlohe-Bartenstein (1760–1811), member of the House of Hohenlohe, whom he married on 22 August 1784.
The couple moved to Germany in 1787, meaning that – on the outbreak of the French Revolution two years later – Gaspard's property in France was seized, he was declared an Émigré, and they were left in the poverty in which he died.