Marcus Garvey Park contains flat lawns and playing fields surrounding the schist outcropping, as well as the Harlem Fire Watchtower, a New York City designated landmark.

[5] The portion of Fifth Avenue in Harlem was also proposed to be renamed Marcus Garvey Boulevard by advocates and the two surrounding Community Boards in 1988, but this effort ultimately failed amid complaints from residents who feared that the street would lose its reputation and lead to confusion.

[6] Before the European settlements, the rocky hill of Manhattan mica-schist was used by the Native Americans as a lookout station to see over the entire island.

During the American Revolutionary War, Hessian soldiers "mounted a battery" at the hill "to command the mouth of the Harlem River".

It was landscaped in 1869, to a plan by the City surveyor Ignaz Pilat,[13] for which the Central Park Commission allocated $15,000.

Walks were graded and the sloping path to the summit was constructed; by the next spring, The New York Times predicted, "croquet playing, decent picnic parties, and a band of music will attract not only the inhabitants of Harlem, but those of New-York".

[14] Blocks of the original rusticated retaining walls, akin to the stonework that surrounds Central Park, can still be seen.

[17] A long vehicular tunnel to send Fifth Avenue through the center of the rock was proposed by Manhattan Borough President Samuel Levy and approved by Robert Moses in 1936.

The watchtower allowed observers to use the natural elevation of the park and the added height of the structure to search for fires, in an era when most buildings were made of wood.