He was a member of the Walderdorff family; the son of Johann Peter von Walderdorff (1575–1635) and his wife Maria Magdalena Greiffenklau zu Vollrads (1595–1678), sister of the Archbishop of Mainz, Georg Friedrich von Greiffenclau zu Vollrads.

[2] His younger brother Johann Philipp von Walderdorff (1620–1689) was also a clergyman, as well as canon in Speyer and Trier.

From 1647 to 1669, Walderdorff was Vicar-General of Mainz, from 1650 dean, later also cathedral provost in the Diocese of Speyer.

Walderdorff acquired in the mid-seventeenth century the Stockheimer Hof on the then Frauenbrüderplatz called Carmelite Square.

The pedestal shows the family crest of Greiffenclau-Vollrads, from which his mother originates, and Walderdorff's Latin inscription "SUB TUUM PRAESIDIUM" (Under your protection may this house stand).