Baker was born in the village of Painted Post in Steuben County, New York.
He spent two years at Overland College before attending Cornell University in Ithaca, New York.
Around 1889 he was appointed as Assistant United States Attorney for the Southern District of New York.
[1] He resigned in 1895[2] to return to private law, but was again appointed to the office several years later and served until 1908.
They had three children, Charles Duane, Jr., David Dudley Field, and Carolyn Beattie.