He was one of the founders of the American Association for the Advancement of Science and served as its President in 1857.

Besides several papers on meteorology in the Reports of the Smithsonian Institution, he wrote The Life of Francis Wayland, a Textbook on Astronomy, and a Memorial of John Barstow (1864).

Through his father, he is a direct descendant of Peregrine White, the first baby boy born aboard the Mayflower while it was anchored at the Massachusetts harbor.

She was a 3rd great-granddaughter of Roger Kinnicutt, who was born in England and emigrated to America around 1635.

They had at least six children: After Esther's death, Caswell married Elizabeth Brown Edmands (1817–1880) in 1855.