[1] His father's summer residence, built in 1791 on the East River opposite Blackwell's Island, was known as" Sunswick" (later known as Ravenswood)[1] and was one of the largest and best appointed private houses around New York.

[9] Joseph, his father, and four of his brothers were painted by Morton H. Bly, which is today owned by the New York Genealogical and Biographical Society.

By the end of 1812, he was commissioned as a captain in Hawkins' Regiment, and promoted to major of the 46th Infantry on April 15, 1814, and resigned at the close of the war.

[14] During his travels north, he began to form his collection of minerals that was considered one of the best in private hands in the country for many years.

From 1827 to 1866, when he declined a re-election, he served as president of the New York Lyceum of Natural History where he was a member for fifty-two years.

Together, they were the parents of four children:[18] In 1829, he purchased around 256 acres (104 ha) of land to build a country seat, known as "Fieldston" (after a family seat in Ireland), on the Hudson River, between the southern part of Yonkers and the Spuyten Duyvil, where he built a lime kiln in 1830, providing him with a large income for several years.