In 1818 it was purchased for the sum of $20,000 by Richard Caton, the husband of Mary, youngest daughter of Charles Carroll of Carrollton, a signer of the Declaration of Independence.

For the last twelve years of his life, Charles Carroll spent his winters in the house, often receiving distinguished visitors there.

The neighborhood underwent a drastic change in the eleven years prior to 1855 and had gone from the wealthiest part of town to the home of poor immigrants.

The citizens of Baltimore protested said plans, and upon his election in 1962, on a platform emphasizing urban renewal, Mayor Theodore McKeldin pledged that the historic Carroll Mansion would be fully restored.

In 1985 the mansion became part of the Baltimore City Life Museums and the collection was expanded to include wallpaper, paint, china, silver, and furniture of the 1820s and 1830s.