The site upon which Madison Square Garden was eventually established was originally occupied by a small passenger depot of the New York and Harlem Railroad.

The site was vacant until 1874 when it was leased to P. T. Barnum who converted it into an open oval arena 270 feet (82 m) long, with seats and benches in banks, which he named the Great Roman Hippodrome where he presented circuses and other performances.

Vanderbilt presented sporting events such as indoor track and field meets, a convention of Elks, the National Horse Show and more boxing, including some bouts featuring John L. Sullivan, who began a four-year series of exhibitions in July 1882, drawing over-capacity crowds.

Barnum also used the Garden to exhibit Jumbo, the elephant he had bought from the London Zoo; he drew sufficient business to recover the $10,000 pricetag.

Vanderbilt eventually sold what Harper's Weekly called his "patched-up, grimy, drafty, combustible old shell" to a syndicate that included J. P. Morgan, Andrew Carnegie, James Stillman and W. W. Astor, who closed it to build a new arena designed by noted architect Stanford White.