He was one of twelve children,[3] including: Morgan Lewis;[4] Gertrude Laura;[5] Julia;[6][7] Alfred; Mortimer;[4][8][9] Susan Mary;[4] Robert James Livingston;[1] Henry Beekman;[1] and Angelica, who married Alexander Hamilton, Jr.[10][11] His paternal grandparents were Robert James Livingston and Susanna (née Smith) Livingston, sister of Chief Justice William Smith and daughter of Judge William Smith.
Lewis's stately home Staatsburgh House in Staatsburg, Dutchess County, New York.
[14] Later, Livingston himself inherited the mansion which he later passed down to his younger daughter Ruth, which the couple used as a summer home and where they raised horses.
[20] Ruth's nephew was Edmund Lincoln Baylies Jr., the lawyer and prominent society member.
[26][27] Through his daughter Ruth, he was the grandfather of Gladys Livingston Mills (1883–1970), who married Henry Carnegie Phipps;[28] Jane Beatrice Mills (1883–1972), who married Bernard Forbes, 8th Earl of Granard;[29] and Ogden Livingston Mills (1884–1937),[30] [31] who would become the 50th United States Secretary of the Treasury,[32] and who married Margaret Stuyvesant Rutherford in 1911.