The double-height main entrance in the cellar, on Edgecombe Avenue, is set in an arched opening with ironwork at its peak.

Numerous African American figures moved into the building, including actor/singer Paul Robeson and musician Count Basie, for whom part of the adjoining section of Edgecombe Avenue is named.

[9] The building is near the top of Coogan's Bluff, a cliff on the western bank of the Harlem River, and faces the Morris–Jumel Mansion to the north and Highbridge Park to the east.

[10] The intersection of Edgecombe Avenue and 160th Street is co-named "Paul Robeson Boulevard" and "Count Basie Place", after two notable residents.

[10][14] The site is part of the former estate of British Army officer Roger Morris, who acquired land in Upper Manhattan in 1765 and built the Morris–Jumel Mansion at the top of Coogan's Bluff.

[7][15] The Morris family lived in the mansion until 1775, and the estate was occupied by the Continental Army, then by British and Hessian officers, during the American Revolutionary War.

The Jumel estate had been split up by the end of the 19th century,[7] and V. K. Stevenson bought the site of 555 Edgecombe Avenue at an auction in 1882.

[17] The construction of the New York City Subway's first line in 1904 spurred the development of row houses and apartment buildings in Washington Heights, which for the first time had easy access to Lower Manhattan.

The first through twelfth stories are clad with beige brick; the window sills, lintels, and band courses are made of terracotta.

[21] The southern and western elevations are made of pale yellow brick with plain rectangular window openings.

There is also a stained-glass skylight in the lobby's ceiling, designed in the Art Deco style, though it was covered up during the 1960s due to fears of nuclear bombings during the Cold War.

[26] In May 1908, contractor Albert J. Schwarzler bought ten land lots on the western side of Edgecombe Avenue from 159th to 160th Street, in exchange for two houses in the Bronx.

[27][28] Schwarzler hired the firm of Schwartz & Gross in 1910 to design a six-story apartment building on the southern half of the site, at 545 Edgecombe Avenue.

Following the completion of 545 Edgecombe Avenue, he again hired the partners to design an apartment building immediately to the north.

[33] The dispute was resolved when the New York Supreme Court ruled that infants should be allowed to use the main entrance on Edgecombe Avenue.

[7] After Schwarzler's death, five of the eleven staff members were fired, and residents were forced to wait up to ten minutes for the only working elevator, prompting complaints from occupants.

[42][43] Following the sale, 32 residents organized a rent strike, claiming that landlord Louis Demburg had interrupted their doorman service, heat, and hot water.

[50] Grace still owned the building when he died in January 1960,[51][52] and Alexander Gross of Eldorado Estates Ltd. announced in March 1960 that he had bought both 545 and 555 Edgecombe Avenue.

[59] Gross resold 545 and 555 Edgecombe Avenue in December 1960 to Matthew Golson, who paid cash and took over mortgage loans that had been placed on both buildings.

[21] By the 1980s, the building had been characterized as rundown, and one resident claimed that the apartments needed repairs and that tenants were falling victim to crimes.

[67] The corner of 160th Street and Edgecombe Avenue, outside the building, was co-named for jazz musician Count Basie and actor/singer Paul Robeson in October 2009 following advocacy from the Harlem Historical Society and the Morris–Jumel Mansion's director.

[13] By the mid-2010s, the lobby was wearing down, and The New York Times wrote that the building looked like it "will sooner or later be consumed by the kind of gentrification that has already remade central Harlem".

[69] After Golson died, the Harkham family bought the building in April 2022 for $26.7 million, just over two-fifths of the original asking price.

[40] Other notable residents included Count Basie,[10][40][72] Andy Kirk,[50][73] psychologist Kenneth Clark,[74][2] composer Duke Ellington,[67][72] writer Langston Hughes,[14] writer/filmmaker Zora Neale Hurston,[14] actress Lena Horne,[14][72] actor/producer Canada Lee,[74] boxer Joe Louis,[40][74] U.S. Supreme Court justice Thurgood Marshall,[70] baseball player Jackie Robinson,[14] and actress Anne Wiggins Brown.