Anthony was born to Quakers on a farm outside the town of Mayfield, New York.

His father died when he was five, and Anthony had to work to support himself, his mother, and his siblings.

Anthony ran a hardware store, was married to Rosa Lyon.

Anthony organized the Seventeenth Independent Battery of Light Artillery; and was commissioned captain when it was mustered into the U. S. Army on August 26, 1862.

[3] As governor, Anthony was known for penny-pinching, and favored programs that did not cost the state much money.