Designed by Cass Gilbert in the Beaux-Arts style, it was erected from 1902 to 1907 by the government of the United States as a headquarters for the Port of New York's duty collection operations.

[11] The building occupies the site of Fort Amsterdam, constructed by the Dutch West India Company to defend their operations in the Hudson Valley.

[12] Gilbert had written in 1900 about his plans for a wide-ranging, site-specific decorative program, which would "illustrate the commerce of ancient and modern times, both by land and sea".

[18][19] Sculptures, paintings, and decorations by well-known artists of the time, such as Daniel Chester French, Karl Bitter, Louis Saint-Gaudens, and Albert Jaegers, embellish various portions of the interior and exterior.

The third- and fourth-story windows, conversely, are less ornately decorated; this was normal for Beaux-Arts buildings, which generally had greater detailing on the more visible lower levels.

[4][16][25] One source described the keystones as representing "Caucasian, Hindu, Latin, Celt and Mongol, Italian, African, Eskimo, and even the Coureur de Bois".

[28] Gilbert planned the Custom House's interior so "all entrances, corridors, stairways and passages [were] arranged on the most direct and simple axial lines".

[55] At the center of the lobby is a three-bay-wide foyer with a pair of round arches to the north and south, which are supplemented by green Doric-style marble columns with white capitals.

[72] About 6,000 square feet (560 m2) of storage space on the ground floor, under the rotunda, was converted into the George Gustav Heye Center's Diker Pavilion for Native Arts and Cultures in 2006.

[82] William J. Fryer Jr., superintendent of repairs of New York City's federal-government buildings, wrote to the United States Department of the Treasury's Supervising Architect in February 1888 about the "old, damp, ill-lighted, badly ventilated" quarters at 55 Wall Street.

[84] The 55 Wall Street building's proximity to the Subtreasury was no longer advantageous, as it was easier to use a check or certificate to make payments on revenue.

"[86] Fryer recommended Bowling Green as his first preference for a new custom house, followed by a site immediately south, along State Street north of Battery Park.

[85] In addition, many local businessmen opposed moving the custom house,[92][93] and a judge ruled in 1891 that the federal government could not take the Bowling Green site by eminent domain as it had proposed to do.

[99][106] The federal government chose an alternate site for the appraiser's warehouse in the West Village of Manhattan,[106] near where much of the city's international shipping activity took place.

[125][126] Some of the opposition centered around the fact that Gilbert was a "westerner" who had just moved from Minnesota to New York City, and several opponents raised doubts about the jury's competence.

[130] Contracts for the building's foundations and structural steel were delayed because the federal government had received several bids, whose estimated completion dates differed significantly.

The government was not allowed to request material from a specific quarry, so multiple contractors submitted bids for numerous types of marble, limestone, and granite.

[161] The U.S. government proposed relocating the Customs Service's administrative offices in 1927 to the Appraiser's Stores Building, but shipping companies spoke out against the move.

[164] During the Great Depression, in April 1937, collector Harry M. Durning commissioned Reginald Marsh to paint murals in the main rotunda as part of the Treasury Relief Art Project, with funds and assistance from the Works Progress Administration (WPA).

[170][171] Durning asked Congress in 1940 to restore the appropriation, saying that "men [were] falling out of ancient chairs, and [...] our valuable records and current papers stacked on desks and improperly filed in decrepit cabinets and bookshelves".

These included increasing cargo-handling and trucking costs; the decline of local railroads; the rapid growth of the southern and southwestern United States and the development of ports in these regions; and the opening of the St. Lawrence Seaway in Canada, which allowed ships to deliver cargo directly to the Upper Midwest.

[175] The building's other tenants at the time included the United States Coast Guard, whose Third District Search and Rescue Command was headquartered on the sixth floor.

[69] With assistance from several organizations and the city government's Office of Lower Manhattan Development, the institute raised $40,000 to conduct a feasibility study of the various plans for the Custom House.

[205] The museum and building were mostly undamaged by the September 11 attacks in 2001, but airborne debris from the collapse of the World Trade Center had to be cleared from some of the interior spaces.

The New York Times said in 1906 that "it is the unity of idea embodied in the new Custom House and enforced by the wealth of sculpture with which it is embellished, more than its mere costliness, that gives to the edifice its unique value".

[31] A Times editorial the same year said that, despite the federal government's initial reluctance to decorate the Custom House lavishly, "few recall the money sunk into stone, bricks and mortar; they enjoy the final touches inside on which millions were not squandered".

[7] After the lower floors were converted into the Heye Center, Benjamin Forgey of The Washington Post wrote that the galleries were "conceived as neutral containers, a sequence of vaulted rooms ingeniously constructed within the old building frame in order not to damage (or indeed even to touch) the original walls".

[70] Stern and his co-authors wrote in the 2006 book New York 2000 that the building, which had been one of the Financial District's "most distinguished white elephants", became a "destination spot" once the Heye Center moved in.

[219] A writer for The Wall Street Journal believed that the museum clashed with "the Custom House itself, which with its newly cleaned ceiling murals depicting ferries and ships seems a bizarre venue for looking at Indian art".

[243] The building has also been used as a filming location for other movies such as Inside Man (2006),[244] How To Lose A Guy In 10 Days (2003), and Black Swan [245] In addition, it has been depicted in commercials for Citibank and BMW.