Colloquially known as "The Castle on the Hill," the building that once housed Music & Art is located in the Hamilton Heights neighborhood of Harlem, in the campus of the City College of New York across the street from St. Nicholas Park.

New York City Mayor Fiorello H. LaGuardia started the high school in 1936, an event he described as "the most hopeful accomplishment" of his administration.

R. O. Blechman, Milton Glaser, Ed Sorel,[4] and Reynold Ruffins[5] – three of the four co-founders of the design firm Push Pin Studios – were M&A students in the 1940s.

Notable graduates from the 1950s included Gloria Davy,[6] Diahann Carroll, Susan Stamberg, Jonathan Tunick, Billy Dee Williams, Peter Yarrow, Tony Roberts, James Burrows, Erica Jong, Felix Pappalardi, and Jeremy J. Shapiro.

The auditorium has excellent acoustics, and features diamond-shaped amber windows that during daylight cast a warm glow on its dark wood interior.

[13] According to the Landmark Commission report, this was not an expensive building for its time, and many of the structural components (like the staircase bracings in the stairwell) were left exposed to save money.

Yet much thought went into humanizing the space and creating a good environment for learning, with plenty of natural light and air, expansive collaborative spaces, and much playful decoration thrown in for good measure: The five- and six-story (plus basement and central tower) L-shaped [building] was designed in an abstracted contemporary Collegiate Gothic style and clad in limestone and mottled buff-to brown iron-spot brick, with large window bays filled with unusual folding-casement steel sash windows.