Altman's was the first big department store to make the move from the Ladies' Mile shopping district to Fifth Avenue, which at the time was still primarily residential.

[2][5][6] The B. Altman and Company Building was designed by Trowbridge & Livingston in the Italian Renaissance Revival style and opened in three phases in 1906, 1911, and 1914.

[10] Marc Eidlitz & Son was the general contractor for the building,[11][12] and Hecla Iron Works manufactured the metalwork.

[16] The structure's facade was generally intended to harmonize with the designs of mansions on Fifth Avenue, which at the time of the building's completion was largely residential.

[19] On Fifth Avenue, the lowest two stories contain a colonnade with double-height engaged columns in the Ionic order, raised upon pedestals and supporting a plain architrave.

In the entrance portico, small stone steps lead to the doors in each bay, which are located underneath glass turtle-shell canopies.

The fourth bay from west contains a metal entrance structure that projects slightly and has a frieze running on top.

To accommodate packages and message deliveries, the building used an extensive system of brass tubing and canvas belting.

[31] An eighth-floor dining room contains ceilings of 40 feet (12 m) as well as a skylight from which the Empire State Building is visible.

[29][27] On the Madison Avenue side of the building, the NYPL occupied an eight-floor condominium spanning 213,000 square feet (19,800 m2) from the 1990s.

[38][39] In addition, Altman did not own any of the land under his Sixth Avenue store; instead, he leased it from two separate sets of owners.

[13] Purchases of property on the block accelerated after plans for Pennsylvania Station and Grand Central Terminal, two major transport hubs nearby, were respectively announced in 1902 and 1903.

[38] By then, some landowners had begun to suspect that the buildings on the block were being sold for commercial purposes, and they either refused to sell or offered only to rent their properties.

[13] In April 1905, Altman received a $4.5 million mortgage loan from the Mutual Life Insurance Company of New York, which covered several properties on the flagship's site.

Three workers were killed and several were injured in a December 1905 dynamite explosion,[52][53] and there was an attempt the same month to sabotage the building's hoisting engines.

[61] Two landowners, Margaret A. Howard and William Waldorf Astor, owned the remaining sites on Madison Avenue.

[23] Trowbridge & Livingston filed plans for the annex in June 1913, which would expand the building's floor area to 900,000 square feet (84,000 m2).

[16] During Benjamin Altman's life, there had never been any exterior signage advertising the store, out of respect to people who lived nearby.

[73] The renovations, in preparation for the 1939 New York World's Fair, involved the removal of the rotunda for additional selling space, as well as new departments designed by H. T.

[76] Rumors of a new structure on the site started circulating in 1970, to which Altman's distributed letters announcing their intention to stay in the same location.

[72] The LPC held hearings to discuss possible landmark status for the building, but Altman's had opposed the designation at the time.

[71][79] The downsizing was required because of New York state legislation that forced the Altman Foundation to divest of some of its business holdings or pay a fine.

[96] In late 1991, KMO proposed that 650,000 square feet (60,000 m2) of the building be converted to the New York Resource Center, a furniture and appliances showroom.

Richard P. Steinberg, one of Olshan's partners, stated in 1994 that three "significant" museums and two educational institutions had expressed interest in the building, though there was no definite commitment.

[25] The renovations, which cost over $170 million, involved restoring many old design elements such as the lobby porticoes, bronze elevator cabs, and cast-iron staircases.

[108] The NYPL's eight-floor condominium was divided four ways in 2012, and the five upper floors were sold that year for $60.8 million to the Church Pension Fund.

[109] The same year, it sold the remaining office condominium unit to Seattle developer Vulcan Inc., headed by Paul Allen, for $93 million.

[15] Altman's had been the first big department store to make the move from Ladies' Mile to Fifth Avenue, which at the time was still primarily residential.

[9][10] Following Altman's example, other major stores made the move uptown to the "middle" portion of Fifth Avenue, including Best & Co., W. & J. Sloane, Lord & Taylor, Arnold Constable & Company, and Bergdorf Goodman.

[27] The running track on the building's roof was used for training by the United States Olympic team, as depicted in the 1981 film Chariots of Fire.