The Town of Phelps is located in the northeast corner of Vilas County along the border with Michigan.

The headwaters of the Wisconsin River are in the northwest corner of the town at Lac Vieux Desert.

Resorts and recreation centered around the town's many lakes are very popular, as are activities such as hiking and camping in the Chequamegon-Nicolet National Forest.

Other events include a large softball tournament held every summer in Wavering Park, and a fireworks display over the lake for Independence Day, in which many people partake from boats.

The area that is today Phelps was originally inhabited by the Chippewa Indian people.

Due to the abundance of healthy forest in the area, the logging business first arrived to exploit this resource in 1896.

WIS 17 runs southwest–northeast across the town, through the unincorporated community of Phelps and to the Michigan border and a connection to Forest Highway 16 near Smokey Lake.

County Trunk Highway E (CTH-E) begins in the Central business district of the unincorporated community of Phelps and runs northwest to serve the area around Lac Vieux Desert before ending at US 45/WIS 32 approximately 2 miles (3.2 km) south of Land o' Lakes.

CTH-K follows the north shore of North Twin Lake and provides a connection to Conover and points west, while CTH-A begins at WIS 17 approximately 1.5 miles (2.4 km) east of the unincorporated community of Phelps and leads south and east to its terminus at WIS 55 in the community of Nelma in Forest County.