His son, the author Fitz Hugh Ludlow, later wrote: my father, mother, and sister were driven from their house in New York by a furious mob.
The front-parlor was full of paving-stones; the carpets were cut to pieces; the pictures, the furniture, and the chandelier lay in one common wreck; and the walls were covered with inscriptions of mingled insult and glory.
'Henry Ludlow wrote that on another occasion he was "mobbed and egged… in broad day light… in the presence of approving & assenting justices of the peace and other officers of the town…" Fitz Hugh also reports that his father was a "ticket-agency on the Underground Railroad."
Henry himself, in one of his few preserved sermons, attacked Great Britain for "her cruel oppression of her East India subjects, often starving... and forced to cultivate opium on land they need to supply themselves with bread…" and defended China "for resisting a traffick which was sapping, by its terrible effects upon her citizens, the very foundation of her empire…" I cannot keep believing that all your sympathies are, irrespective of treaties, on the side of these men, who to secure their inalienable rights, breathed the very spirit and performed the very deeds of the Heroes of '76.
May I not say that had God one hour before the deed blanched their skin, and straightened these woolly locks the united world would have applauded them, and our own Countrymen hailed their arrival here as the very men they delighted to honor.