His paternal great-grandfather, Timothy Woodruff (1732–1788), served as a Captain during the French and Indian War.

His maternal grandfather and maternal great-grandfathers all served as Captains in the American Revolutionary War: Solomon Woodruff (1763–1822), Noah Woodruff (1730–1790) and Eliphalet Curtis (1736–1806).

He returned to Connecticut in 1841 and settled in Bristol, where he worked in a clock factory until 1845, when he moved to New Haven.

He was elected a member of the common council in 1848 and served several terms.

They had a daughter, Harriet Jane Woodruff Ward (1865–1937), and a son, Timothy Lester Woodruff (1858–1913), a three-term Lieutenant Governor of New York.