In 1845 Frazer entered the United States Military Academy at West Point, and graduated four years later standing 34th out of 43 cadets.

[1] Frazer had spent his U.S. Army career on "routine garrison duty at various points" across the United States,[2] until resigning his commission, which was accepted on March 15, 1861.

[1] On April 5 was ordered to Louisiana to supervise the regular army recruiting for Baton Rouge and for New Orleans, Frazer's headquarters.

Frazer chose to resign from the Confederate Army in March 1862, but re-entered the service as colonel and commander of the 28th Alabama Infantry on November 2.

[1] His most notable military service occurred after he was appointed a brigadier general on May 19, 1863,[6] and given command of the 5th Brigade of the Army of Tennessee that July.

[8] Burnside sent a brigade under Col. John F. DeCourcy to deal with Frazer's men, while the rest of the Union soldiers quickly marched on and captured Knoxville, Tennessee, on September 2.

Frazer thought his Confederates to be surrounded, outnumbered, and lacking enough provisions needed for a prolonged siege, and he unconditionally surrendered his garrison.

[8] Frazer was sent to join other captured Confederate officers at Fort Warren in Boston Harbor, where he spent the rest of the war.

Due to public and political criticism of his actions at the Cumberland Gap,[9] Frazer's appointment was promptly rejected on February 16, 1864.