While studying at university, Withers was inducted into the Freemasons, an organization he would remain with for life, specifically as a leader in the Knights Templar (Freemasonry).

[2] During the American Civil War, he entered the Confederate States Army in early 1861 as the major of the 18th Virginia Infantry Regiment.

Withers is largely responsible for preventing destruction of Danville's infrastructure (bridges, rails) by Confederates forces toward the end of the war.

[6] President Grover Cleveland appointed Withers as the United States consul to British Hong Kong, from 1885–89, when he resigned.

[7] His extensive collection of personal papers are housed with the Kegley Library of Wytheville Community College.

He was also a cousin to Thomas W. Chinn, Member of the United States House of Representatives from Louisiana and the US Minister to the Kingdom of the Two Sicilies.

Withers was a distant relative of figures such as George Washington and Robert E. Lee, as well as a direct descendant of Nicolas Martiau, founder of Yorktown, Virginia.