The town was first settled in 1789 by Ezekiel Merrill and his family who were transported there from Bethel, Maine, in canoes managed by members of the local Pequawket tribe.

The first saw-mill was built on the East Branch of the Ellis River by Col. Thomas Poor in 1791 and was used to provide the lumber for the still standing Merrill-Poor House.

As the town developed, local factories made doors, sash, blinds, starch, cheese, boots, shoes, edge-tools, carriages, and harness.

Located on an intervale surrounded by mountains, Andover is noted for "its grand and beautiful scenery," with streams abounding in trout.

After the Civil War, the town became a popular destination for sportsmen, especially fishermen attracted by the nearby Richardson Lakes.

It was the location of the first live transatlantic television signal broadcast by satellite Telstar 1.

The river is home to many fish and wildlife, from which avid hunters and fisherman try their luck.

It borders the towns of Roxbury and Byron to the east, Rumford to the south, Newry to the southwest, Andover West Surplus to the west, and Andover North Surplus to the north.

The racial makeup of the town was 98.8% White, 0.4% Native American, 0.4% Asian, and 0.5% from two or more races.