The building consists of a two-story base and an eight-story upper section, with a facade of green and white marble.

These spaces are decorated extensively with marble and aluminum, and the outer and elevator lobbies also include the Goelet family's crest.

Prior to the construction of the present office building, Mary and Ogden Goelet lived in a mansion at 608 Fifth Avenue.

The mansion was torn down in 1930, and Ogden's nephew Robert Walton Goelet built the office building on the site, which was finished in 1932.

[4] From the outset, 608 Fifth Avenue was designed as a commercial structure that would maximize the rapidly rising land value of the area, with retail on the lower floors and office stories above.

To maximize the surface area of these show windows, Faile designed the third and higher stories on two-story-tall columns that are recessed 5 feet (1.5 m) from the facade.

[4][13][19] Faile engineered the building's columns to be strong enough to accommodate a possible conversion of the light court into additional office space.

Other decorative elements include a monogram consisting of interlocked letters "G", as well as the Goelet family crest of a swan.

[18] The original main entrance had a bronze door with the Goelet family's crest and reportedly cost $14,000 (equivalent to $322,649 in 2024).

[20] Various marbles are used, including what The New York Times described as "aurora rossa, samosa golden, American pavonazzo, bleu belge, numidian red and Belgian black".

The elevator doors themselves are decorated with stylized leaves atop vertical bands of white and yellow metal.

[19][21] The construction of Rockefeller Center made it difficult to forecast whether numerous small stores or a single large retailer would be more suitable for the site.

[22] When the second story was built, it was cantilevered from the third-story slab instead of being supported by columns above the first floor, thus maximizing the first-floor retail space.

[9] In 1920, Robert Walton Goelet commissioned the construction of an art gallery at 606 Fifth Avenue, directly south of Ogden's estate.

[9] Rockefeller Center's developers allowed Robert Walton Goelet to keep the lots at 2–6 West 49th Street because the company considered his "interest and concern" to be a significant factor.

[35][36] In June 1930, Robert Walton Goelet hosted a ceremony to give craftsmanship awards to 23 workers who were involved in the project.

[37] The building was completed by 1932 but, due to a lack of interest from large tenants, the ground-floor space was divided into smaller units.

[38] Within the area bounded by Sixth and Fifth Avenues between 48th and 51st Streets, the Goelet Building was among the few plots that was not owned outright by Rockefeller Center's developers by the end of 1932.

[15] These included brokers Cowan & Co.,[40] restaurant operator Susan C. Palmer,[41] the Tecla Pearl Company,[42] and a Manufacturers Hanover Corporation bank branch.

[53] The building's tenants in the 1940s included watch dealers Louis Manheimer & Brothers[54] and gem importers Lieberman & Bienenfeld.

[15] Tenants during the 1950s included the Jewelry Industry Council,[58] wool-trade group Woolens and Worsteds of America,[59] the Institute of Boiler and Radiator Manufacturers,[60] and airline Avianca.

[64] The Fifth Avenue Association, a local civic group, recognized the Pakistan International Airlines office as the area's "best storefront alteration" during 1962–1963.

[66] 608 Fifth Avenue became the Swiss Center Building in 1964 when fourteen Swiss-owned enterprises formed a coalition to "foster commercial, cultural, travel and financial activities identified with Switzerland".

[7][78] The leasehold was sold to RFR Holding, a company held by German investors, in 1998 for about $22 million, though Korein retained ownership of the land.

[66][93] The deal generated $70 million for Vornado, which at the time was experiencing financial losses due to the COVID-19 pandemic in New York City.

[94] By 2021, the empty storefront was being used to advertise HBO Max's release of the film No Sudden Move;[95] it also hosted New York Fashion Week the same year.

[22] Christopher Gray, writing for The New York Times in 1990, referred to 608 Fifth Avenue as "one giant Art Moderne cigarette case of marble".

[101] The New Yorker architectural critic Lewis Mumford sarcastically described 608 Fifth Avenue as "an excellent period reproduction—Modernique, 1925", regarding it as little more than a parody of the earlier Childs Restaurant building.

[6] Robert A. M. Stern, in his book New York 1930, called the building "a luxuriously detailed but bastardized interpretation of the International Style".

[13] In 2017, architectural historian Anthony W. Robins wrote that the Goelet Building's lobby was "one of New York's best-kept Deco secrets".