Hoyt was born in Stamford, Connecticut on July 24, 1823, the sixth child and son of Joseph Blachley Hoyt (1787–1854) and Mary Blachley Weed (1794–1854), who were married in 1813.

Hoyt was a trustee of a fund of $250,000 that had was raised by private subscription in 1881 for the benefit of Grant and his family, to insure the General an income for the remainder of his life.

After leaving the Senate, he returned to run the Hoyt leather merchant business, which his sons took over after his death.

[3] Together, they were the parents of eight children: Hoyt was injured when he was thrown from his carriage in Greenwich, Connecticut and broke his collar bone.

[19] His grandson, Edward Kenneth Hoyt (1885-1938), who attended the Browning School in New York City and the Lawrenceville School and was a 1908 Yale graduate, who married Maria Augusta Daurer DeWust, the daughter of Francisco S. Daurer and Maria E. Kalhwoda, on June 5, 1926, in Paris.