The historians Eric Foner and Stephen Middleton found that his mother was a fair-skinned mulatto daughter of Judge Thomas Heyward, Jr., a signer of the Declaration of Independence, and his father a wealthy young white man, whose family rejected their relationship.

Because of his appearance and high proportion of European ancestry, Miller could have passed for white in the North, but chose to identify as black and return to the South to help the freedmen.

Receiving a scholarship, Miller attended Lincoln University, a historically black college in Pennsylvania, where he graduated in 1872.

African-American political rivals dismissed him as a white imposter attempting to take advantage of the post–Civil War black electorate.

As African-American candidates competed in "black" districts, men's ancestry became part of the political fodder; tensions became heightened between mulattoes like Miller and darker-skinned politicians such as George W. Murray.

He was also a delegate to the 1895 South Carolina constitutional convention called by the white supremacist governor, Benjamin Tillman.

Historian Michael Perman wrote that "in no other state was a single public figure identified so vividly and indisputably with disfranchisement.

The new constitution was one of a number passed in southern states at the turn of the century that were designed to effectively disfranchise African-American citizens by changes to voter registration rules.

South Carolina's version required longer residency, literacy tests administered by white supervisors, poll taxes, and $300 worth of property.

[9] The Democrats passed the constitution and effectively disfranchised most black voters for more than half a century, which also disqualified them from serving on juries or holding local office.

Miller did gain the support of Tillman to establish a land-grant college for African Americans in the segregated state school system.

Supporting United States participation in World War I, he helped recruit 30,000 black men to the Armed Services.