The first permanent settlement occurred in 1819 at Whitesville, although an attempt was made to settle the town in 1798.

[3] From November 16, 1896, through June 1936 Whitesville was served by the New York & Pennsylvania Railway (and predecessors) as part of its Canisteo to Ceres (via Greenwood, Genesee, Oswayo and Shinglehouse) main line.

Following severe floods in July 1935 the railroad was partially out of service, and its abandonment in June 1936 was readily approved by the Interstate Commerce Commission which from an early date approved railroad abandonment to promote highway truck use.

The loss of railroad service did much to accelerate economic decline of the Whitesville area.

According to the United States Census Bureau, the town has a total area of 34.5 square miles (89 km2), all land.

New York State Route 248 is a major east-west highway in the town that intersects with New York State Route 248A (Cryder Creek Road) west of the community of Whitesville.