[4] His maternal grandfather was Count James-Alexandre de Pourtalès, a prominent banker and art collector who served as chamberlain to the King of Prussia Frederick William III.
[6][7][8] Ganay, and his elder brother were both featured in James Tissot's 1868 group portrait painting The Circle of the Rue Royale.
Coleraine Vansittart, Baron Rodolphe Hottinguer, Marquis Alfred du Lau d'Allemans, Comte Alfred de La Tour-Maubourg, Charles Haas (who was a source for the character of Charles Swann in Marcel Proust's Remembrance of Things Past).
[5] Like his father and grandfather before him (who owned the famous Comte de Pourtalès Collection which was auctioned off in 1865), he was an art collector.
[5] Through his son Jean, he was a grandfather of Count Bernard de Ganay, who married Magdeleine Goüin.