Hughes was born in about 1832 to a Euro-American plantation owner and an African-American slave mother in Charlottesville, Virginia.

In 1844, Edward McGee, a wealthy Mississippi planter, purchased young Hughes as a Christmas gift for his wife.

[3] Though Hughes did not work daily in field labor, he did have bad experiences with Edward's wife, Madam McGee.

Matilda was also unable to produce breast milk to nourish her children due to the brutality that the Madam had consistently inflicted upon her body.

"[3] Three of his escape attempts ended with severe beatings, the scars of which he bore for the rest of his life, he said, on his body and soul.