Pickman's father Col. Benjamin Pickman, Sr.,[5] one of the most important merchants in Salem, had been a Loyalist, his estates confiscated by the Colonial government and was forced to flee America for England, only returning to Salem in 1785 after the end of the Revolutionary War.

Benjamin Pickman Jr. also served in the Massachusetts Senate in 1803, as well as a member of the executive council of the State in 1805, 1808, 1813, 1814, and 1819–1821.

They were the parents of Anstiss Derby Rogers, who married merchant William Shepard Wetmore on September 5, 1843.

The daughter of Sherman, Georgette Wetmore Sherman, married Harold Brown (Rhode Island financier), son of John Carter Brown and grandson of Nicholas Brown Jr. Benjamin's niece, Mary Toppan Pickman, married Massachusetts Congressman and diplomat George B. Loring, who is Benjamin's great-nephew through his brother Clark.

His aunt, Judith Pickman, married physician and scientist Edward Augustus Holyoke.

This article incorporates public domain material from the Biographical Directory of the United States Congress