Numerous tenants have occupied various parts of the house through the years, including Samuel Dexter, Christopher Gore, John Jeffries, Harrison Gray Otis, Anna Ticknor's Society to Encourage Studies at Home, and temporarily in 1824, the Marquis de Lafayette.
Shortly after the house was built, its owner Thomas Amory met financial trouble and subsequently sold the property.
... On the evening of August 30, 1824, Lafayette held a reception in his apartments at the Amory house; and this function was attended by many prominent ladies of Boston.
Around 1885 it was "remodeled ... with 2-story Queen Anne-inspired oriel windows of black-painted pressed metal and fanciful dormers on the Park Street roofline"[7] and "a set of black metal shop fronts that reach out and down to the falling sidewalk..."[8] By 2008, "the once great mansion stands barely recognizable, although the basic brick volume and Adam entrance portico with fanlight and curving granite steps (one half is missing) are more or less intact.
2014 renovations of the building yielded original eastern white pine interior sheathing boards, which were re-milled by a nearby reclaimed lumber company.