The village was developed in the early-to-mid 19th century, and has retained the character of that period better than many small communities in the state.

Early development was focused in the area around Kidder Hill Road, while in the second quarter of the 19th century the village center gradually migrated northward, anchored by the construction of the Congregational Church in 1834 and the relocation of its post office in 1841 into what is now the town hall.

Grafton's building and economic good times ended about 1870, with little new development taking place until the 20th century, when tourism provided a significant boost to the economy.

Its rough southern and northeastern boundaries are the two branches of the Saxtons River, extending east of the north branch along Chester Road and Route 121 East, and extending a short way westward along Houghtonville, Middletown, and Hinckley Brook Roads.

Very few of the historic buildings have been altered in unsympathetic ways, such as the application of synthetic siding or the installation of vinyl windows.