During the Pike's Peak Gold Rush he briefly relocated to Colorado where in November 1859 he founded the town of Breckenridge.

Receiving a promotion to colonel, he led the regiment from September 11, 1863, until his resignation on July 5, 1865.

[3] Elected as a Republican to the United States Senate upon readmission of Alabama to the Union, Spencer served from July 13, 1868, to March 3, 1879.

[5][6] He was appointed a commissioner of the Union Pacific Railroad with help from his previous leader, Major General Dodge.

[7] The couple then spent two years on a ranch in Nevada tending to mining interests before settling in Washington, D.C., about 1880.