The town's first secondary school was established by Alexander Hyde, son of its first minister, in his house on West Park Street.

Townspeople not long thereafter banded together to form the private Lee Academy on a parcel of land above the town center.

It was designed by James Bryning and constructed of locally quarried marble, a response to several high-profile fires in the town, at a cost of $30,000.

The original 1895 building is sited at the end of Academy Street, and the 20th-century additions, located to its south and east, do not intrude on the view.

The original slate roof was replaced in 1994 with asphalt shingles, at which time the building was also equipped with gutters and downspouts.

A corridor connecting the two sections was added in 1976, when fire code necessitated the removal of some wooden stairs from the main building.