In the late 19th century, Upper Manhattan was still primarily farmland and marsh dotted with occasional small houses and taverns- a bucolic setting that would soon begin to disappear with the arrival of the Industrial Revolution.

The clean air and remoteness of the area soon attracted newly created millionaires who built homes that were extravagant monuments to display their wealth.

[5] The centerpiece of Paterno's seven-acre (2.8 ha) estate, located in what is now the Hudson Heights neighborhood was a 35-room four-story Neo-Gothic mansion in the form of a castle.

The main reception hall was 80 feet (24 m) above street level, off of which were Paterno's den and a parlor, music room, and library.

An antique clock in the entrance hall triggered chimes on the hour and half-hour in the castle's tower and, at certain times of the day, operated a $7,000 organ on the second floor gallery.

When the castle was slated for demolition, the organ was purchased and donated to St. Paul's Episcopal Church in Glen Cove, New York.

[8] The area was becoming increasingly residential, and The New York Times quoted Paterno as saying that "the many improvements in that part of the city...had led to a strong residential movement in that area with a definite demand for the finer type of garden type apartments.”[1] Paterno died in 1946 aged 69, eight years after the castle's demolition.

It was commissioned by Cleveland Walcutt, an engineer, who built it on land he purchased from the estate of the editor of the New York Herald, James Gordon Bennett Jr.

The living room and balcony afford views of the downtown Manhattan skyline, as well as the George Washington Bridge and the Hudson River.

On May 12, 2005, the wall partly collapsed, producing a landslide that buried the northbound lanes of the parkway and six parked cars.