Located at 1st and A streets NE in Washington, D.C., on the eastern slope of Capitol Hill, the site's first building was a red brick tavern and hostel called Stelle's Hotel, built around 1800.

In August 1814, during the War of 1812, the British burned the nearby United States Capitol building.

The building was actually financed by Washington real-estate investors, who had heard rumors that some members of Congress were considering relocation of the national capital in the aftermath of the burning.

South Carolina Senator and former Vice President of the United States John C. Calhoun, who had been a leading member of the Fourteenth Congress when it met in the Old Brick Capitol, died in the boarding house in 1850.

The government sold the Old Capitol Prison in 1867 to George T. Brown, then sergeant-at-arms of the U.S. Senate, who modified the building into three rowhouses collectively known as "Trumbull's Row."