Ladenburg was born to a wealthy Jewish family on 22 August 1822 in Mannheim, Grand Duchy of Baden.
His uncle, Leopold Ladenburg, and his wife Delphine, were close friends of Johannes Brahms.
On 17 March 1852, Ladenburg married Eugénie Adèle Halphen (1829–1866), the daughter of a respected Parisian family related to the Paris Rothschilds.
Together, Eugenie and Emil were the parents of four children, two sons and two daughters:[4] Ladenburg died on 8 January 1902 in Frankfurt, then part of Hesse-Nassau.
Ladenburg hosted Johannes Brahms at his home when the "Joachim Quartet" played there in November 1894.