After the war, Saxton was a strong advocate for the enfranchisement of African Americans and served as the Freedmen's Bureau's first assistant commissioner.

His father attempted to secure a place for Rufus Saxton at Brook Farm in West Roxbury, Massachusetts, a transcendentalist community started by George Ripley and attended by Nathaniel Hawthorne.

[2] As the Civil War broke out, Saxton served as a quartermaster and ultimately a brigadier general for the Union forces.

During the war, he commanded the Union defenses at Harpers Ferry and he was awarded the Medal of Honor for his "gallant service" there in May and June 1862.

[7] As the military governor, he directed the recruitment of the first regiments of black soldiers called the First South Carolina Volunteer Infantry (33rd USCT) who fought in the Union army.

Saxton recruited Gullah men from the surrounding South Carolina Sea Islands and other contrabands from Georgia and Florida.

[13] Saxton later served as assistant commissioner for the Freedmen's Bureau, where he pursued the policy of settling freed slaves in land confiscated from white landowners in the Sea Islands,[citation needed] until he was removed from his position by President Andrew Johnson.

According to an account by his close personal friend, author Thomas Wentworth Higginson, Saxton "had been almost the only cadet in his time at West Point who was strong in anti-slavery feeling, and who thus began with antagonisms which lasted into actual service.

Saxton also spoke in Congress against widespread confiscation of firearms owned by African-Americans, stating such actions were "clear and direct violation of their personal rights" as described in the Second Amendment.

[16] Saxton appointed his friend, author and abolitionist Thomas Wentworth Higginson, colonel of the 1st South Carolina Volunteers, the first official black regiment.