[4] Miller was the namesake of his grandfather, the famous abolitionist, businessman, and philanthropist Gerrit Smith.

He grew up on the family's estate in Peterboro, New York, helping his grandfather by hiding escaped slaves in a barn or attic.

[1]) He enrolled in Harvard in 1865, but set back by health problems, left before graduating;[5]: 2  in 1924 the university awarded him a honorary Master of Arts degree.

[10] Miller's health failed after the burning on March 2, 1936, of his home, built by his great-grandfather, Peter Smith, in 1803.

[11] Fortunately, Miller had already, in 1928, given to the Syracuse University Libraries his grandfather's huge collection of correspondence, business records, daybooks, and pamphlets.