Designed by Skidmore, Owings & Merrill (SOM), the mixed-use development contains condominium apartments, office space, and retail shops.

Upon Olympic Tower's completion, architectural writers such as Ada Louise Huxtable and Christopher Gray criticized its design.

[8][9] Olympic Airways, the national airline of Greece, subleased 647 Fifth Avenue immediately north of the store in 1965[10] and opened a sales office there the next year.

The tower's developers, Arlen Realty and Olympic Airways president Aristotle Onassis, received a zoning variance that exempted them from having to construct a setback on 51st Street.

[30][31] Under city laws regulating privately owned public spaces, Olympic Tower's owners were obligated to provide a minimum number of trees, light fixtures, benches, movable chairs, and planters.

[12][19] The designs of the rooms are relatively simple and, because the full-height windows did not allow mechanical systems to be embedded in the facade, each apartment has central ventilation.

[39] An Associated Press writer called the condos "very European in concept", as most bathrooms had bidets and a hallman delivered mail directly to each apartment.

[20] One of the duplexes was occupied by Adnan Khashoggi and was built with its own swimming pool,[40][41] as well as five bedrooms, six bathrooms, indoor gardens, a 300-seat catering kitchen, a ballroom, and a sauna.

[43] The penthouse apartments were the highest in the world when Olympic Tower was completed, although the uninhabited domes of the Waldorf Astoria New York were slightly taller.

[48][52] Because of his relatively small amount of experience in office design, Arlen had initially been hesitant to hire Lapidus, who specialized in residential buildings.

Meshulam Riklis, chairman of McCrory's parent corporation Rapid America, recalled that the slab would be topped by a glass "cube" with a sky plaza, a full-story art gallery, offices, and executive apartments.

[56][57] McCrory's president Samuel Neaman said at the time that the site of Best's Fifth Avenue flagship was worth more as a potential real estate development than as a retail location.

[59] Shortly after the closure of Best's was announced, an editorial appeared in The New York Times, specifically naming Lapidus's involvement in Olympic Tower as an example of Fifth Avenue's decline.

[61] In February 1971, New York City mayor John Lindsay proposed the district as a means to preserve the retail character of Fifth Avenue's midtown section.

[16] Olympic Tower's developers hired a graphics design firm to create a sign with propellers that spelled the building's name three times.

[44] Due to high demand for the condos, additional apartments were created within the existing space shortly after Olympic Tower was finished.

[16][28] Arlen hired public relations company Burson Marsteller,[39] which advertised the building in Europe, South America, and Mexico.

[71] Arlen sold its interest in Olympic Tower to Williston S.A., a Panamanian company that was owned by the Onassis family, in late 1979.

[75][76] As a result, New York City Planning Commission head Robert F. Wagner Jr. threatened to revoke Olympic Tower's certificate of occupancy in 1979, saying the building's owners had not upheld an agreement to add stores to the retail mall in exchange for zoning bonuses.

[83] Olympic Tower's condominium units continued to see high demand from Europeans looking to live on Fifth Avenue, though critics said the residences were not well maintained.

One brokerage executive said the building's operators "want a tenant in keeping with H.Stern, Mark Cross and Versace", which already occupied retail space on the block.

[39] In May 2012, the Alexander S. Onassis Public Benefit Foundation sold a 49.9 percent stake in the commercial portion of the tower and three neighboring structures to real estate investment firm Crown Acquisitions for $420 million.

[88][89] The owners received a $760 million commercial mortgage-backed security (CMBS) interest-only loan in May 2017 from Deutsche Bank, Goldman Sachs, and Morgan Stanley.

These included General Electric's board of directors, who worked at Rockefeller Center, as well as designers Laurice Rahmé, Giuseppe Zanotti, and Lorraine Schwartz, who preferred the structure for its history, architecture, and proximity to several high-end stores.

[98] The choreographer, director, and dancer Ron Field also lived in the building,[99] as did musician Roger Waters,[39][100] actor Nicolas Cage,[39] and the wealthy Wertheimer family of Israel.

"[39] After SOM's plans for the building were announced, a writer for Newsday said Olympic Tower would provide "a clear test of strength between God and Mammon", being right across from St. Patrick's Cathedral.

[54] According to a writer for the website CityRealty, Olympic Tower's design was "more conservative, yet paradoxically also more cutting-edge" compared to the Galleria condominiums on 57th Street, which was built around the same time.

[25] When the building opened, Paul Goldberger of The New York Times criticized the facade as being "not as refined as Skidmore Owings and Merrill's best work".

[16][38] Goldberger called the building's architecture "oppressively banal"[106] and described it as "overwhelm[ing] Fifth Avenue like an aircraft carrier beside a row of sailboats".

[107] Ada Louise Huxtable of the same newspaper said Olympic Tower was "about as nondescript as anything that size can be"[30][108] and described the residential lobby as "mildly offensive in design and taste and quite disposable".