When the processing facilities were moved to Ellis Island in 1892, Castle Garden was converted into the first home of the New York Aquarium, which opened in 1896 and continued operating until 1941.

In the 1940s, New York City parks commissioner Robert Moses proposed demolishing Fort Clinton as part of the construction of the nearby Brooklyn–Battery Tunnel.

Late that year, Lieutenant Colonel Jonathan Williams of the United States Army Engineers began planning a series of fortifications in New York Harbor.

[16] Designed by John McComb Jr. with Jonathan Williams as consulting engineer,[17][18] West Battery was roughly circular with a radius of approximately 92 feet (28 m).

[21] A 1816 proposal to construct two small office buildings at Fort Clinton was canceled due to public opposition, and the castle lay dormant for three years.

[13][29] Over the years, the fort hosted other political figures such as U.S. presidents Andrew Jackson,[30] John Tyler,[29] and James K. Polk,[29] as well as Hungarian governor-president Lajos Kossuth.

[63][64][a] The majority of immigrants processed at Castle Garden were from European countries, namely Denmark, England, Germany, Ireland, Italy, Russia, Scotland, and Sweden.

[49] The New York state government's Board of Emigration Commissioners had been established in 1847 to operate medical facilities and a registration center for immigrants.

Although the board had acquired the Marine Hospital on Staten Island soon after its establishment, their efforts to open a registration center were unsuccessful for several years.

[59] In 1864, to convince immigrants to enlist in the United States Armed Forces during the American Civil War, the County Bounty Committee erected a recruitment center next to Castle Garden.

[81][82] Two years later, the Board of Emigration Commissioners constructed a one-story labor exchange building, a waiting room, and an information office, and they made repairs to Castle Garden.

[55] By then, The New York Times wrote that the surrounding Battery Park was "a haven for the 'runners' who approached innocent Irish and German newcomers, offering them nonexistent lodgings for their money".

[47][97] In the aftermath of the fire, several city officials proposed shuttering the Castle Garden immigration center and restoring the fort as a venue for "public enjoyment".

[105] For example, in 1885, the state government refused to allocate $10,000 for repairs to the depot's ferry dock because the city technically owned Castle Garden.

[108] Robert Chesebrough, a businessman who owned several structures around Battery Park, had also advocated for the closure of the Castle Garden processing depot.

[110] The federal government notified New York state officials in February 1890 that it would take over immigrant-processing duties at Castle Garden within sixty days.

[117][118][119] The state's Commissioners of Emigration had forbidden the federal government to continue to use Castle Garden until the Ellis Island immigrant depot was completed.

[147] For instance, the skylights on the roof acted as a greenhouse that raised the temperature of the water in the tanks, and the saltwater fish in the aquarium were dying off because of the low salinity of the Hudson River.

The plans included constructing a third story for workrooms and laboratory space, installing tanks behind the fort, adding a new mechanical plant in the basement, and covering the facade with a gray cement finish.

[193] George McAneny, a former mayor and the chairman of the Regional Plan Association's board, proposed restoring Castle Garden;[194][195] he continued to advocate the fort's preservation for nine years.

[206] The Fine Arts Federation of New York held an architectural design competition in August 1942, soliciting plans for a renovation of Castle Garden.

[207][208] Despite ongoing disputes over the fort's fate,[208] workers began removing metal from Castle Garden on September 25, while the rest of the building remained in place for the time being.

[218][219] Both the House and the Senate approved the legislation,[220][221] and president Harry S. Truman signed the bill into law on August 12, 1946, enabling the United States Department of the Interior to determine whether to take over the fort.

[229] After Interior undersecretary Oscar L. Chapman indicated in August 1947 that Congress would allocate money to the project in 1948, the board voted to delay further action for one year.

[257][258] According to the NPS, Castle Clinton typically has over three million visitors a year, making it one of the most visited national monuments in the United States.

[228] In 1962, New York City parks commissioner Newbold Morris proposed relocating 18 columns from the soon-to-be-demolished Pennsylvania Station to a promenade outside Castle Clinton.

[279] The work included restoring the exterior and interior walls; adding a shingle roof; removing a moat and other facilities related to the fort's use as an aquarium; and repairing officers' quarters, parade ground, and ammunition storage areas.

[32][254] The NPS planned to spend $1.5 million to replace two structures, add exhibitions, restore the roof and parade ground, and reconstruct a doorway that had been sealed in 1974.

[294][295] By 1996, the Conservancy for Historic Battery Park was raising $350,000 (equivalent to $680,000 in 2023) for a seasonal tensile structure, to be placed above the fort between April and October of each year.

[300] That December, the NPS erected a tent with seven body scanners at Castle Clinton, where visitors to the Statue of Liberty National Monument underwent a security screening.