His father was a theatre actor and entrepreneur who owned the New Orleans Gas Light Company.

At the same time, they bought an Italianate style cottage in Newport, Rhode Island on the southeast corner of Kay and Ayrault Streets that formerly belonged to Boston merchant Caleb Chace.

[1] In honor of his wife, who died in 1867, he funded the construction of the Sisters Mary and Elizabeth Hospital Louisville, which opened in 1874 shortly before his death.

[17] Through his daughter Lina, he was a grandfather of Baron Waldemar von Zedtwitz, a bridge champion who never married.

The John La Farge stained glass widows in the house which had been commissioned by the Caldwell sisters, were saved by Bishop James Edwin Cassidy for installation in the convent of St. Patrick's Church at Fall River.