[1] Built as an inn and tavern around 1770 in the Georgian architectural style in red brick with white wood trim by Henry Fite (1722–1789), the building became known as "Congress Hall" when it served for two months as the new nation's seat of government in 1776–77.

The "Henry Fite House" served as his home and office during the next 20 years in the 1820s and 30s, where he directed his growing wide-ranging business, financial and investment empire, which by mid-century had made him the richest man in America.

Later during the decades following his 1827 death, his sons and family cut up and divided the estate building large elaborate substantial townhouses on a grid of streets including Peabody's Institute added north of original colonial era in Baltimore Town.

The Maryland Society of the Sons of the American Revolution placed a large, elaborate, polished bronze memorial tablet in front of the "Henry Fite House" on February 22, 1894, describing the building's service to the nation.

[2][3] When the Civic Center (now the Royal Farms Arena) was built on the site, the bronze tablet of 1894 was preserved and mounted on the outside wall facing Hopkins Place, near the northeast corner of the building.