Born in St. Albans, Vermont,[1] Withey read law to enter the bar in 1843.

[2] Withey was nominated by President Abraham Lincoln on March 10, 1863, to the United States District Court for the Western District of Michigan, to a new seat authorized by 12 Stat.

He was confirmed by the United States Senate on March 11, 1863, and received his commission the same day.

His service terminated on April 25, 1886, due to his death in San Diego, California.

[2] President Ulysses S. Grant nominated Withey to the United States Circuit Courts for the Sixth Circuit on December 17, 1869, and he was confirmed by the Senate on December 22, 1869, however he declined the appointment.