After graduating with a bachelor's degree from Harvard University in 1958 he began his career as an editor at The Atlantic Monthly.
In 1963 he became the first publisher of The New York Review of Books; a post he maintained until his retirement in December 1986.
He thereafter worked as managing partner of the publishing firm The Lakeville Journal Company.
His first marriage to writer Sallie Bingham ended in divorce.
[1] From 1976 to 1978, Ellsworth served as chairman of Amnesty International USA, devising a method of greatly increasing the organization's size and capabilities by setting up a direct-mail fund-raising operation.