Details about Hayter's early life and upbringing are scarce; however, she does reflect in her diary on 12 August 1842 that: What a strange position mine has been as a child.

Though separated from my mother when very young I treasured her memory with idolatrous love though often my heart felt broken with what seemed to be her forgetfulness of me and I believed her to be everything that a christian ought to be - alas the bitterness this disappointment cost me God only knows.

[5] By her early 20s Hayter was undertaking voluntary work at the General Penitentiary at Millbank for prison reformer Elizabeth Fry.

She was selected by the British Ladies' Society for the Reformation of Female Prisoners to be matron on board the convict transport ship Rajah.

Besides the advantage which the prisoners derived from the instruction given by the clergyman, they were also under the superintendence of a female of superior attainments, who had previously been an officer at the General Penitentiary…[7]The Ladies of the Convict Ship Sub‑Committee supplied the convicts "with various articles of clothing, besides haberdashery, materials for needle‑work, and knitting, (in order to afford employment during the voyage) and with books of instruction, comprehending that most blessed book whose value the Committee are anxious should be rightly appreciated — the Holy Scriptures.

She would recall the romance in her diary in June 1842: My beloved Charles this day twelthmonth in a walk with you on the poop of the Rajah you told me to make an entry in my journal to this effect that my fate was linked with yours and you bid me look at it at some future time to see if it were so…[12]Three days after Hayter's arrival in the colony, Lady Jane Franklin invited her to join her at dinner and questioned her about Millbank Penitentiary.

Franklin reflected "She [Hayter] has evidently strong sense, with most amiable and pleasant manners, with an intelligent countenance though plain.