He led the expansion of the company from "a modest, family-controlled business to an industry giant with interests extending into broadcasting and baseball.
The young Sargent attended the private St. Mark's School and a year at Harvard College before enlisting in the Navy during World War II.
Sargent remarried on December 21, 1985, to Elizabeth Nichols Kelly, the fiction and books editor of Cosmopolitan magazine.
He ranged from editing the poetry of Theodore Roethke to publishing bestsellers by Stephen King and others; in the 1970s, he recruited Jackie Kennedy Onassis as an editor.
In the summer of 1972 his former wife Neltje Doubleday Kings led a shareholder effort to take the company public, but it was defeated.
[5] While Sargent served as president and CEO until 1978, he led the company through a major expansion, expanding its publishing and diversifying its businesses.
As reported by Bruce Weber, By 1979, the year after he left the presidency and was made chairman, Doubleday was publishing 700 books annually.
Deeply involved in its social life, he was described as a socialite and for years hosted a Christmas Eve party strictly for single people.