The academy was built by a hanging tree where a British spy was executed in 1777; his bones were discovered and relocated during construction of a dormitory in the 1860s.

[1] The Ford Administration Building is the last significant structure remaining from the academy and it was added to the National Register of Historic Places in 2006.

[6] During the 1960s, the New York Jets football team used the Academy and its playing field as a summer training camp.

A yellow brick road made of Dutch pavers, which used to lead from the steamboat dock to the academy and of which only one section remains as part of a parking lot, is thought by city historian John Curran, among others, to be the inspiration for the Yellow Brick Road in alumnus L. Frank Baum's Oz books including the Wizard of Oz.

[7] Notable alumni include actor Lee Marvin, actor Guy Williams, banker Sandy Weill, writers L. Frank Baum and Paul West, politician Chauncey Depew, civil liberties advocate Narciso J. Alegre, and Olympic swimmers Carl Robie and Stephen Rerych.