The town and village are named after Judge Ambrose Spencer.

The Village of Spencer is located in the center of the town and is south of Ithaca.

It was called "Drake Settlement," at first, but the name was changed to "Spencer" when the town was organized and the village assumed the neighboring community of "Milltown."

The village was popularly called "Pumpkin Hook" in the past.

The First Presbyterian Church was listed on the National Register of Historic Places in 2005.

28.5% of all households were made up of individuals, and 7.8% had someone living alone who was 65 years of age or older.