Constructed between 1909 and 1912 as a private residence for businessman James Buchanan Duke and his family, the building has housed the New York University (NYU)'s Institute of Fine Arts since 1959.

The house generally contains long windows and high ceilings and has a portico in the middle of the 78th Street facade.

The interior of the first floor is designed in the French Classical style and consists of four large corner rooms, used as classrooms, which surround a main entrance hall.

[9] Cook intended the block to house first-class residences, not high-rises, and only sold lots for the construction of private dwellings.

[3][13] Built for the family of James Buchanan Duke, it has served as a building for the New York University (NYU)'s Institute of Fine Arts since around 1959.

[3][14] Trumbauer drew heavily upon the design of French architect Etienne Laclotte's Château Labottière [fr], built in 1773 in Bordeaux.

[4][22] To make the Duke mansion appear as an overscaled version of the Château Labottière, Trumbauer hid the service rooms in the basement and the servants' bedrooms in the attic.

[4] The house generally contains long windows and high ceilings to give it the impression of a large mansion.

The spaces between the windows on both stories are made of stone panels in low relief, while a band course separates the floors horizontally.

The former music room has arched openings with swags, fluted pilasters with capitals, relief panels, and an architrave with festoons and a cornice; many of the decorations are gilded.

[13][14] The kitchen and servants' dining room was placed in the western side of the basement, while the laundry was in the northeast corner.

[41] James and his brother Benjamin Newton Duke moved the company's headquarters to New York City in the 1900s.

[39] By the middle of the decade, James Duke was worth $50 million, owned four properties across the eastern U.S., and lived at Benjamin's house at 1009 Fifth Avenue.

[54] In September, Duke sold a small parcel on the north side of the site, measuring 20 by 100 feet (6.1 by 30.5 m), to his neighbor William Payne Whitney.

[18][56] The 1910 United States census records James and Nanaline Duke as living in Benjamin's house at 1009 Fifth Avenue.

[63] The New York Times dedicated a page in an illustrated supplement to photographs of the house, which it dubbed the "costliest home opened on Fifth Avenue within a year".

[66][c] The 1920 United States census showed that all of the servants working at the house at that point had been hired after 1915, except for their 50-year-old cook Mathilda Andrews.

[17][67] Nanaline did spend significant amounts of time at the 78th Street house, but James preferred to live in his other homes after World War I.

[67] At the end of World War I, the Duke family obtained the Rough Point estate on Newport, Rhode Island, in an attempt to ingratiate themselves with New York City's high society.

[68] James Duke also obtained the Lynnwood house in Charlotte, North Carolina, and lived there in the 1920s, just before his death.

[71][72] Doris wished to avoid the formality of placing the residence through an auction, which would entail having guests enter the home to assess the furnishings, potentially damaging it.

[72] As a teenager, Doris continued to reside in the family house on 78th Street, referring to it as "the rock pile" in her adulthood.

This prompted Nanaline Duke to ask the New York Supreme Court in 1929 to reduce the house's valuation from $1.6 million to $970,000, citing the apartment construction.

[77][78] When Doris Duke turned twenty-one years old in 1933, she received a substantial part of the bequest that had been held in trust for her.

[81][82] Even when Doris Duke remarried to Porfirio Rubirosa in 1947, she retained ownership of the 78th Street house and several other properties.

[26][86] NYU's occupancy of the Duke House preserved it for the time being, especially when other mansions on Fifth Avenue's "Millionaires' Row" were being demolished.

[88] The New York Landmarks Conservancy praised the Duke House's "superb adaptive reuse" when the NYU renovation was completed.

[89] The LPC designated the house as an individual landmark on September 15, 1970, calling it "one of the adornments of Fifth Avenue and one of the last reminders of the Age of Elegance".

[1] The same year, the LPC designated the house as part of the Metropolitan Museum Historic District, a collection of 19th- and early 20th-century mansions around Fifth Avenue between 78th and 86th Streets.

[86] In 2014, NYU proposed creating an enclosed breezeway slightly above the alley separating the Duke House and 3 East 78th Street.