[1] The original structure was designed by Starrett & van Vleck and constructed by the Cauldwell-Wingate Company, with numerous other engineers and contractors.

[4] The architects created a modern interior for the department store and followed a new city zoning law requiring setbacks for buildings' upper floors for Saks' administrative offices.

[5]: 4 The 36-story 623 Fifth Avenue was designed by a partnership between Lee Harris Pomeroy Associates, hired by Swiss Bank Corporation (the initial building owners), and Abramovitz Kingsland Schiff (staff architects for Saks).

The presence of twin entrances, while relatively rare for department stores, emphasizes the building's size and full-block Fifth Avenue frontage.

[3]: 91 The main element of the facade is on its second and third floors – a 14-bay-wide order of fluted pilasters supporting an architrave, all constructed of Indiana limestone.

The eighth story is relatively bare, while the ninth has narrow windows surrounded alternatively by slender colonnettes and cast stone rectangular panels.

[1] The tower and original Saks building are easily viewable from across the street at 30 Rockefeller Plaza and its Top of the Rock observation deck.

[1] The lobby originally had a waterfall and 1989 Richard Serra work, Fin, made of curved oxidized iron and weighing 18 tons.

He worked as a peddler and paper boy before moving to Washington, D.C., where at the age of only 20, and in the still-chaotic and tough economic times of 1867, only two years after the United States prevailed in the American Civil War, he established a men's clothing store[16] with his brother Isadore.

[17][18] A. Saks & Co. occupied a storefront in the Avenue House Hotel building at 517 (300–308) 7th Street, N.W., in what is still Washington's downtown shopping district.

[19][5]: 2  Andrew Saks ran the New York store as a family affair with his brother Isadore, and his sons Horace and William.

[5]: 2  Horace Saks wanted to move to the Fifth Avenue shopping district, which had been first developed in 1905 with the opening of the B. Altman and Company Building at 34th Street and which was gradually expanding northward.

[5]: 2 Saks & Company leased the Buckingham Hotel and Belgravia Apartments, on Fifth Avenue between 49th and 50th Streets, in April 1920 for $35 million.

[5]: 4–5 [27][28][29] Gimbel took over the Herald Square lease, paid off Saks' debt, and bought $8 million of the company's stock.

Gimbel, wife of the company president, designed elegant clothes and introduced women's culottes to the American public.

[42] By 1938, Fortune magazine characterized the Fifth Avenue store as upscale, as opposed to "the anthill bargain basement tables on Herald Square", the latter of which ended up closing in the 1960s.

The flagship's first escalators would be installed within the expansion, as Saks officials sought to minimize disruption to the main store.

[6]: 557 [46] These plans brought criticism from observers who worried the expansion would lower the quality of Saks' flagship to that of Bloomingdale's or Macy's.

[6]: 558 [47] The lots to the east had been proposed as the site of a tower since the 1950s but had not been developed because of a lack of participation by Saks real estate interests.

[48] Work on the expansion proceeded before the tower was approved and, in November 1979, a set of new escalators opened in the rear of the Saks flagship.

[6]: 558 [49] Project architect Hambrecht Terrell then renovated the second floor with a brick corridor containing shops for notable fashion designers.

The largest point of contention was the tower addition's 50th Street frontage, which Saks wanted for display windows, but this was ultimately allocated to Swiss Bank for its entrance.

Pomeroy took over as main architect after Abramowitz Kingsland Schiff's plan was criticized as Brutalist and out-of-context with surrounding buildings.

[61] The tower was able to obtain a prestigious Fifth Avenue address because the skyscraper is on the same land lot as the 1924 store and was built using its air rights.

[8] Architectural writer Paul Goldberger criticized the design as bland, saying the Swiss Bank Tower was "so successful at not offending that it ends up having nothing to say".

[63] The original flagship and the tower addition have a single owner, initially Swiss Bank Corporation, which had leased the ground from Saks for 100 years.

A year later, Saks' new CEO Fred Wilson announced plans to undo much of the renovation and spend $150 million to further remodel the flagship.

[65] In August 2007, the United States Postal Service began an experimental program selling the "plus" ZIP Code extension to businesses.

The company president desired reinventing the flagship, opening up its floors and closing its Cafe SFA to create a new eatery, the Parisian-style L'Avenue.

The lowest ten stories of the tower would continue to be part of the Saks Fifth Avenue flagship, while the 11th and higher floors were to become 172 apartments, each with an average area of about 2,000 square feet (190 m2).