However, his grandfather Charles Carter (1732–1796), a patriot during the American Revolutionary War (and member of the House of Burgesses and the Virginia House of Delegates), dissipated the wealth inherited from his paternal grandfather Robert Carter I.

Both his younger brothers who had children married women from Fredericksburg, and survived the Civil War and are buried in St. Louis, Missouri.

Frank Carter (1813–1896) was a steamboat captain who transported mail, freight and passengers on the Ohio River and was buried in Louisville's historic Cave Hill Cemetery like his father.

Walker Randolph Carter Jr. {1807–1889) included CSA Major Frank Carter (b.1838) who enlisted as a student at the University of Virginia, survived the conflict, and after the war became a respected St. Louis businessman, and was buried like his father in historic Bellefontaine Cemetery in St.

In 1862, Commander Carter led the first iron-hulled steamer on the Great Lakes, the USS Michigan, which was the object of Confederate subversion, but its officers remained loyal to the Union.