In 1871, he became a member of the Committee of Seventy in New York City, which was instrumental in breaking up the Tweed Ring, and later assisted in the prosecution of the indicted officials.

In the retrial of the General Fitz-John Porter case, he obtained a reversal of the decision of the original court-martial.

In politics, he allied himself with the Republican Party on its organization, being a frequent speaker in presidential campaigns, beginning with that of 1856.

He never held political office, although he was a candidate for the Republican U.S. senatorial nomination for New York against Senator Thomas C. Platt in 1897.

Ambassador to the United Kingdom to succeed John Hay in 1899, and remained in this position after Theodore Roosevelt's ascendancy to the presidency until the spring of 1905.

[10] On October 16, 1861, he married Caroline Dutcher Sterling Choate (1837–1929), who had been born in Salisbury, Connecticut.

Caroline was an artist and an advocate for women's education, helping to establish both Brearley School and Barnard College.

[12] Joseph and Caroline were the parents of five children, two of whom predeceased their parents:[13]: 683 The family owned a large country house, known as Naumkeag, which was designed by Stanford White and is today open to the public as a nonprofit museum in Stockbridge, Massachusetts.

[15] His funeral was held on May 17 at St. Bartholomew's Church in New York, where it was attended by the British Ambassador, Sir Cecil Spring-Rice, the French Minister of Education, M. Hovelacque, and the Assistant Secretary of State, William Phillips, among many others.

Memorial services were held on May 22 in London, England, and on May 31 at Trinity Church on Wall Street.