The southeast corner of Fifth Avenue and 52nd Street was planned as a hotel in the early 1900s after the Roman Catholic Asylum vacated the site.

Other nearby buildings include 650 Fifth Avenue to the west, 660 Fifth Avenue to the northwest, Austrian Cultural Forum New York to the north, 12 East 53rd Street and Omni Berkshire Place to the northeast, St. Patrick's Cathedral to the south, and the International Building of Rockefeller Center to the southwest.

The facade along Fifth Avenue is three bays wide and, at the ground floor, contains a large opening surrounded by blocks of rusticated limestone.

[27] At the second and third stories on 52nd Street, the central three bays form a slightly projecting pavilion, with an ornate balustrade in front of the second-story windows.

The second-story windows have individual balustrades at the bottom and are flanked by engaged columns that support lintels with denticulation.

The third-story windows are topped by keystones, while a cornice with modillions, dentils, and brackets runs above the fourth story.

[27][32] Following a 2016 renovation, the Cartier store was expanded to 44,000 square feet (4,100 m2) on five floors, with an interior stairway connecting all the stories.

[36] The second floor contains salesrooms for fine jewelry,[38] in addition to a hospitality suite with a private dining room.

[44] Edward Holbrook bought a 50-by-100-foot (15 by 30 m) lot on the south side of 52nd Street, just east of Fifth Avenue, from Worthington Whitehouse in August 1900.

[52] That October, Chisholm's syndicate filed plans for an 18-story apartment hotel, designed by William C. Hazlett, to be built on that site.

[14][54] Holbrook canceled plans to build his apartment hotel in August 1902 after agreeing with the Vanderbilts to restrict their respective lots to private residential development.

[60][61] Holbrook did build a 5-story residence on the eastern site of the lot at 8 East 52nd Street, but he sold it to Ernest Kempton Adams.

[62][63] Gilbert designed a six-story residence for Holbrook on the western side of his 52nd Street lot, which would have been the site of the apartment hotel.

[65][66] Plant's residence at 651–653 Fifth Avenue was completed in 1905, and he and his wife Nellie moved into the house;[22][67] the structure had cost $400,000 to construct.

[70] In January 1906, the Holbrook House hosted the wedding of their daughter Lilian to Count Guillaume de Balincourt.

[75] 4 East 52nd Street was then occupied by the family of real estate developer Harry James Luce, who had moved into the house by August 1910, when he received a mortgage on the property.

[76] Immediately to the west, Fifth Avenue was widened in 1911, and the marble steps in front of Plant's house had to be cut back.

[80][83] When the neighboring house at number 647 was sold to an art dealer the next month,[84] the Real Estate Record and Guide said the sales marked "another step in the transition of this section of Fifth avenue from the residential to the business stage".

[86][87] The trade consisted of $100 in cash and included a double-stranded necklace of 128 flawlessly matched natural pearls valued at the time at $1 million (equivalent to $23,781,800 in 2023[b]).

As part of the Plant House's conversion, the original front doorway in the middle of the facade on 52nd Street was removed.

[92] Nicholas C. Partos took a 63-year lease on Luce's house in March 1927 and considered erecting a 12-story commercial structure on the site.

[8][95] The house was resold shortly afterward to the 653 Fifth Avenue Corporation, which owned the former Plant residence that Cartier occupied.

[96] 4 East 52nd Street came to be occupied by the French Chamber of Commerce and the Alliance Française de New York, which respectively elected Pierre C. Cartier as their president in 1935 and 1938.

The buyer, who reportedly paid for the buildings in cash, held the properties as an investment and continued leasing 653 Fifth Avenue to Cartier.

[110] Aristotle Onassis—the president of Olympic Airways, which at the time had a sales office at 647 Fifth Avenue—established a family trust called Victory Development in March 1970.

Victory formed a joint venture with Arlen Realty & Development Corporation to acquire Best's store, 647 Fifth Avenue, and the Cartier Building.

[113] As part of the Olympic Tower's construction, a pedestrian plaza was built east of 647 Fifth Avenue and the Cartier Building.

[126] The Cartier Building received another renovation in 2014, designed by Beyer Blinder Belle in conjunction with Thierry W. Despont.

[36][129] Guy Trebay of The New York Times described the 2010s renovation as having turned the Cartier Building into a "thing of rational and distinctly Gallic beauty".

[129] A writer quoted in Bloomberg News stated that the new interior was the "single biggest signifier of wealth in Manhattan".