Originally located on Orange Street, it adopted its nickname, "The Academics", in acknowledgment of its close association with Yale University.

[5] For many years, Hillhouse served not only New Haven but also suburban towns around the city that did not have high schools of their own.

[7][8] Hillhouse became involved in athletic competition as early as 1866, when some boys formed a club to play a sport that is described as having "resembled rugby and soccer."

By 1884, students were participating in several sports, including modern football, which had been invented by Walter Camp of New Haven.

Team competition in baseball, tennis, ice hockey, indoor polo and yacht racing also had been established around this time.