Charles Evans Hughes III and Gordon Bunshaft of Skidmore, Owings & Merrill (SOM) designed the building, along with Roy O. Allen and project manager Patricia W. Swan.

510 Fifth Avenue was built as a bank for the Manufacturers Trust Company, whose president Horace C. Flanigan wanted the design to be inviting to customers.

Walker & Poor was hired in 1950 to modify the original proposal but were replaced with consulting architect SOM after Bunshaft convinced bank executives that a complete redesign would be cheaper.

508 Fifth Avenue, originally a rowhouse, became a commercial structure in the early 20th century, housing tenants such as the Gotham Silk Hosiery Company and the candy store Huyler's.

[15][17] According to Horace C. Flanigan, the CEO of Manufacturers Trust when the building was constructed, the design was intended to convey a "sense of stability and security" while using modern materials.

[18] A press release, issued when the branch opened in 1954, said that the design would advertise itself; Bunshaft likewise compared the high-capacity layout to that of a store.

[27] By leasing 508 Fifth Avenue, and thus taking its air rights, Salmon had obtained a taller height for his skyscraper than would be normally allowed under zoning codes.

[23] According to architecture critic Lewis Mumford, the objections to using an all-glass facade were alleviated by the fact that the site was mostly surrounded by skyscrapers.

[55] Facing Fifth Avenue is a 7-foot-wide (2.1 m), 16-inch-thick (410 mm) steel bank vault door, built to designs by Henry Dreyfuss of the Mosler Safe Company.

[62] The original escalators were parallel to the Fifth Avenue facade and were designed as freestanding diagonal structures, giving the appearance that they had no supports of their own.

[48] In front of the western wall, Bertoia was commissioned to create a "floating" or sculptural screen of 800 intersecting brass, copper, and nickel panels.

[71][72] Manufacturers Trust began negotiating in 1941 with the Mutual Life Insurance Company of New York, which owned both the Ziegler Building and 508 Fifth Avenue.

[72] By October 1944, Mutual Life agreed to clear the land,[26] and the New York State Banking Department approved Manufacturers Trust's proposed relocation that December.

[76] Additionally, after the Ziegler Building's tenants were given notice that their leases would be canceled upon the end of January 1946,[77] they successfully filed a lawsuit to enjoin their eviction.

[71] Early the next year, in January 1951, Mutual Life had evicted the tenants of 508 Avenue and the Ziegler Building, giving both properties to Manufacturers Trust.

[30] SOM may have also gotten the commission through one of its principals, Louis Skidmore, who was on the board of directors for another Manufacturers Trust branch,[80][81] and who was friends with Flanigan, the bank's CEO.

[80] Charles Evan Hughes III's winning design featured glass walls and a plainly visible bank vault.

[84] The opening of 510 Fifth Avenue attracted attention from many media outlets, which praised various aspects of the design, such as the glass facade and the vault's visibility from the street.

Chase kept the 150,000 square feet (14,000 m2) of air rights over the building, retained ownership of the Bertoia sculptures, and continued to maintain a branch on the first floor of 510 Fifth Avenue.

[102] The project would create two new retail spaces on Fifth Avenue by cutting new doorways into the facade, rotating the escalators away from the windows, and reducing the vault wall.

[8] Preservationists sued to stop the changes,[103] and a New York Supreme Court judge issued a temporary order blocking the renovation in July 2011,[104][105] which was subsequently converted into an injunction.

[32] The escalators were replaced, the wall of the former vault was removed, and additional steel framing and reinforced polymer fabric was installed to provide structural support.

[62] Joe Fresh occupied 14,000 square feet (1,300 m2) of retail space on the ground floor and second-floor mezzanine for its flagship store, starting in 2012.

[114][115] Clothing retailer The North Face signed an eight-year sublease from Joe Fresh in 2016, occupying 20,000 square feet (1,900 m2) on the basement, first, and second stories.

[112] Vornado announced in July 2023 that it would sell 510 Fifth Avenue and three other structures,[118][119] and the Reuben Brothers finalized their purchase of the four buildings that August, paying $50 million.

[8] The AIA Guide to New York City called it "a glass-sheathed supermarket of dollars" and stated that the building "led the banking profession out of the cellar and onto the street".

[35][39][23] The building's glass design has been called a "metaphor for honesty and transparency in banking" and a "symbol of a self-confident era" which influenced commercial architecture.

The criticism in Harper's was mainly based on the fact that the building still had a significant amount of air rights remaining from the lots at 510–514 Fifth Avenue.

[127] The LPC first held hearings to grant landmark status to the building's exterior in 1985 and 1986, but the property was not designated because of opposition at the time from Manufacturers Hanover and Mutual Life.

[2] In designating the interior as a landmark, Commission chairman Robert B. Tierney said that the building's "luminous ceilings, spacious floor plans, white marble piers and other minimalist features blur the distinction between inside and out".