Following the American Civil War, he became a conservative Democratic political activist at a time when most freedmen joined the Republican Party.

Once Elias gained freedom, he embarked on a public speaking career in which he took up the cause of the Democratic Party and spoke on behalf of former Confederates and slaveholders.

[1] Within a year of his birth, the Polk family, along with those they enslaved, relocated west to the Duck River Valley of Middle Tennessee.

At this time, Elias's rental contract was transferred to a "Mr. Matthews" at the Nelson's House Hotel in Columbia, Tennessee.

"[6] At a meeting whose speakers included Arthur St. Clair Colyar and Henry S. Foote in June 1867, Polk called for harmony between blacks and whites.

Historian Zacharie W. Kinslow states, "Elias Polk understood three things following the American Civil War.

[1] After returning to Nashville, Polk traveled back to Washington, D.C., a few years later, where he met President Grover Cleveland three days before his death.

He was informed by Captain Samuel Donelson, an employee of the U.S. House of Representatives, that Elias was to be reappointed to his old position as a "laborer" at the Capitol.

[2] Historian Zacharie Kinslow states, "During his life, Elias Polk went from being enslaved in the White House to one of the most controversial African-American political activists of his day.

"[1] For journalist Jesse J. Holland, the author of The Invisibles: The Untold Story of African-American Slaves in the White House, "In today's parlance, we'd call (Polk) an 'Uncle Tom' because he's taking the gentry's side.