He was a member of the Walderdorff family; the son of Johann Peter von Walderdorff (1575–1635) and his wife Maria Magdalena Greiffenklau zu Vollrads (1595–1678), sister of the Archbishop of Mainz, Georg Friedrich von Greiffenclau zu Vollrads.

[2] His younger brother Johann Philipp von Walderdorff (1620–1689) was also a clergyman, as well as canon in Speyer and Trier.

[4] He became Imperial Privy Councilor and served as Reich Vice Chancellor for 11 years.

Walderdorff acquired in the mid-seventeenth century the Stockheimer Hof on the then Frauenbrüderplatz called Carmelite Square.

The pedestal shows the family crest of Greiffenclau-Vollrads, from which his mother originates, and Walderdorff's Latin inscription "SUB TUUM PRAESIDIUM" (Under your protection may this house stand).