His twin brother was Henry Leavitt Ellsworth, first Commissioner of the United States Patent Office.

As an Anti-Jacksonian Ellsworth was elected to the Twenty-first, Twenty-second, and Twenty-third United States Congresses and served from March 4, 1829, to July 8, 1834, when he resigned.

[7] In 1847, Elsworth became judge of the Connecticut Supreme Court and served from 1847 to 1861, when, by the constitutional provision relative to age, he retired.

He twice declined to accept the nomination to the United States Senate, and retired from public life.

[8] The lawyer and orator Rufus Choate said of Ellsworth before the Massachusetts General Assembly: "If the land of Shermans, Griswolds, Daggets and Williams, rich as she is in learning and virtue, has a sounder lawyer, a more upright magistrate, or an honester man in her public service, I know not his name."