Built beginning in 1805, with a number of later alterations, it remains a prominent example of Federal architecture in New York City.

It was designated a National Historic Landmark in 1974[2] for its association with Matthew C. Perry, commandant of the adjacent Brooklyn Navy Yard 1841–1843, whose opening of Japan to the West in 1854 revolutionized trade and international affairs.

The former Commandant's House is set on a bluff overlooking the western side of the Brooklyn Navy Yard, a few blocks south of the East River.

The building has an extensive history of alteration, but the interior, its original main block, retains Federal period architectural details, including leaded front entry sidelight windows, carved wooden paneling, and wooden floorboards (although the latter have been covered by other flooring).

Perry's signature achievement was his 1854 expedition to Japan, in which he compelled the previously cloistered nation to open its borders more widely, inaugurating a series of economic and geopolitical changes.