[8] In 1989, he married art historian Louisa Wood Ruby,[3] who has written about the drawings of Paul Bril[10] and Jan Brueghel the Elder,[11] and who worked at The Frick Collection for many years.

Starting in 2010, Ruby has worked on editorial projects for Station Hill Press in Barrytown, New York, including co-editing with Sam Truitt Eating the Colors of a Lineup of Words: The Early Books of Bernadette Mayer.

[2] Ruby has co-written memoirs about the Supreme Court with his great uncle Milton Handler,[17] and he edited the writings of his half-brother David Herfort, a poet who served in jail and then died in a car accident in Spain at the age of 22.

[18] Other notable relatives of Ruby include his stepfather, chemist Eli M. Pearce;[19] his aunt, foreign-policy expert Antonia Handler Chayes.

They are also, in their broken way, up-to-date, streetwise.”[1] Jerome Rothenberg wrote on the back cover of trilogy Memories, Dreams and Inner Voices: “His project here—to explore ‘the varieties of unconscious experience’ as they come to him—is an aspect of what Gary Snyder once described as ‘the real work of modern man: to uncover the inner structure and actual boundaries of the mind.’”