It was demolished and replaced with the elaborately decorated headquarters of Freedman's Savings Bank, established by Congress in 1865 for recently emancipated enslaved people and freedmen.

The third and present building to be constructed on the site was the Treasury Annex, designed by Cass Gilbert in the Beaux-Arts style and completed in 1919.

The initial plan of eventually expanding the Treasury Annex to H Street NW never happened due to several factors.

During the 19th century, elegant houses built for prominent individuals were constructed along the western, northern, and eastern sides of Lafayette Square.

On the corner of Madison Place and Pennsylvania Avenue, Dr. James Gunnell built a five-story house in 1831 that was later rented to government employees.

[1][2] Real estate prices increased considerably after the Civil War, and some houses on the square were converted to office space or rental properties.

[4] Land just east of the square facing Pennsylvania Avenue had already become a popular location for financial institutions, owing to the fact the Treasury Building stood across the street.

[1] The combination of mismanagement and fraud with the Panic of 1873 resulted in the failure of Freedman's Savings Bank, which had 37 branches in Washington, D.C., and 17 states.

"[6][9][10] Deposit insurance did not yet exist in the United States, so only around half of the bank's customers could receive part of their lost funds, which often took decades.

[11][12] The bank building sat empty until it was purchased by the federal government in 1882 and used as office space for the Justice Department and Court of Claims.

[1][6] In 1902 the McMillan Plan was created to change the design and layout of the monumental core and park system in Washington, D.C.

"[1] Despite aspects of the McMillan plan being implemented in other parts of the city, changes to Lafayette Square did not occur until World War I.

[14] On September 27, 1917, an Act of Congress provided funds for the building's construction and allowed the Treasury Secretary to "secure special architectural and expert services.

Senator Frank B. Brandegee said if the new building had "no artistic finish at all", it would be a "public calamity" since it would stand across from the "finest example of pure Grecian art.

The second was the U.S. Chamber of Commerce Building, also designed by Cass, that was built in 1925 on the northern side of H Street across from the Decatur House.

[1][19] On November 8, 1964, the Treasury Annex was added to the newly created District of Columbia Inventory of Historic Sites.

The legacy of Freedman's Bank also serves as a reminder that we must continue striving for greater financial inclusion for all Americans – particularly those in underserved and minority communities – so that they can share in the benefits of our growing economy.

The ceremony included remarks from Secretary Lew, civil rights activist and politician Andrew Young, and businessman Alden J. McDonald Jr., president of Liberty Bank and Trust.

[13] Since the original plan was for the Annex to eventually be expanded north to H Street, the first floor facing Pennsylvania Avenue was built higher.

The basement level includes offices, mechanical rooms, and a tunnel connecting the Treasury Building across the street.