He was the youngest son of seven children born to Lewis Morris Rutherfurd (1816–1892) and Margaret Chanler Stuyvesant (1820–1890).
[1] Rutherfurd was a direct descendant of Peter Stuyvesant, the last Dutch Director-General of New Netherland before it became New York,[5][6] as well as John Winthrop, the first Governor of Massachusetts.
[2] Rutherfurd was one of the social elite nicknamed "The Four Hundred" by Ward McAllister, a number supposedly taken from the capacity of Caroline Webster Schermerhorn Astor's ballroom.
The approximately 38-room home was located alongside his brother Stuyvesant Rutherfurd's property in Allamuchy Township, New Jersey.
Besides the mansion, construction included gardens, a boathouse, a swimming beach, a hydroelectric powerhouse, a 9-hole golf course, and kennels.
[12] After his second wedding, in 1920, Rutherfurd built an estate called Ridgeley Hall in Aiken, South Carolina, where he pursued his favorite hobby of breeding fox terriers.
[18] Rutherfurd then remained a bachelor until age 40, reportedly having numerous affairs with married socialites, including Ava Astor.
[39] Only weeks before their wedding, which took place on February 11, 1920, Lewis, Rutherfurd's oldest son, died of pneumonia.