Born in Elizabethtown, New Jersey, Caldwell was two-years-old when his mother, Hannah, was killed by British troops passing through their farm.

Caldwell graduated from the College of New Jersey (now Princeton University) and studied law with the said Elias Boudinot until his move to the District of Columbia.

[1] Caldwell worked as a lawyer in Washington, D.C alongside Francis Scott Key for a number of years.

Both men were organizing members of the American Society for Colonizing the Free People of Color in the United States.

"[3] At the age of twenty-four, in 1800, Caldwell was appointed clerk of the Supreme Court at Washington and held this post until his death in 1825.