A member of the wealthy Ephrussi family, he spent the first ten years of his life in Odessa, a major port on the Black Sea where his grandfather was a grain industrialist, before moving to Vienna.

An account of the collection hanging in his study appears in a letter written in 1881 by the Symbolist poet Jules Laforgue (later published in La Revue blanche).

But, to the distress of some of the Impressionists, he continued to buy other types of art, including pictures by his friends Gustave Moreau and Paul Baudry.

It also was at this time that he began to collect Japanese lacquers and netsukes, the subject of Edmund de Waal's The Hare with Amber Eyes (2010) which also devotes considerable attention to Charles' life and artistic interests.

He was one of the inspirations for the figure of Swann in Marcel Proust's À la recherche du temps perdu (In Search of Lost Time; titled Remembrance of Things Past in the first translation).