After the American Revolution, voters attempted to rename the town "Bradford", "Middletown", "Liscomb", and finally "Orange".

The large quantities of yellow-orange ochre found in Mount Cardigan may have been the source of the name Orange.

The highest point in Orange is the summit of Mount Cardigan, at 3,155 feet (962 m) above sea level, near the eastern edge of the town.

In the northern part of town, the South Branch of the Baker River has its source, flowing north toward Groton and Dorchester.

The northeastern corner of the town drains north via Atwell Brook to the Cockermouth River in Groton, a tributary of Newfound Lake.

The eastern edge of town drains off Mount Cardigan via several brooks leading to the Fowler River in Alexandria, another tributary of Newfound Lake.

The southeastern part of town drains to Mill Brook, a southward-flowing tributary of the Smith River.