His family played important roles in the history of the United States during the colonial revolution and civil war periods.

[4] The remains of the family home and gardens still exist on the banks of the Ashley river in South Carolina at Middleton Place.

[9][8] He resisted solicitations to join the Confederacy choosing to stay with the Union in the Pacific West citing obligations to the Navy and Government.

This decision strained family relationships and on February 22, 1865, in his absence Middleton Place was burned to the ground by a detachment of the New York volunteer regiment.

[14] Together, they were the parents of: Middleton died on April 27, 1883, in Washington, D.C.[19] His wife was quoted in his obituary as saying, "upon no subject was he uninformed and his erudition was remarkable, and as a linguist he was one of the finest in the country".