[1][2] He was the son of Ezra Ludlow (1786–1861)[3] and Rachel Seguine (1788–1864),[4] who married in Staten Island on February 6, 1808.

His paternal grandfather, Obadiah Ludlow (Ludlam),[8] an officer of the New Jersey troops during the American Revolution,[9] was George Washington's right hand man throughout the War.

[12] During that session, Ludlow was elected speaker of the Assembly with 85 votes against 39 for Jeremiah Ellsworth, the Whig candidate.

[10] During his time as speaker, Governor Horatio Seymour vetoed the "Bill for Extension of the Basis of Banking Capital".

They were the parents of six children, including:[19] In 1889, he sold "Oakdale Farm", a 400-acre (1.6 km2) estate, to Frederick Gilbert Bourne.