However, by around 1880 "without any change in the legal appellation 'House of Industry,' that term has come to be understood as designating its penal character.

"[3] An article in the national Frank Leslie's Sunday Magazine (1884) described the prisoners on Deer Island in the 1880s: "they in the main are from the lowest stratum of the cosmopolitan society of New England's metropolis, embracing representatives of almost every nationality under the sun, and from the shortness of the sentences, many being confined for 10 days only, for nonpayment of one dollar and costs for drunkenness, and none for more than a year.

[2] In 1991 about 880 inmates were transferred permanently to the Suffolk County House of Correction in Boston's South Bay area.

In Sylvia Plath's novel The Bell Jar, the protagonist, Esther Greenwood, visits Deer Island Prison.

In Dennis Lehane's novel Mystic River, the character Jimmy Marcus served two years in Deer Island Prison in Winthrop.