[1] Fairfield's town center was laid out in the 17th century by its founders, who included Roger Ludlow.

The area was divided into Four Squares, one for Ludlow, one for a minister, one for civic buildings, and one for a town common.

Elements of this early division survive in the layout and placement of civic and religious buildings.

The village center was burned in 1779 by British troops during the American Revolutionary War, resulting in the destruction of all of the town's civic buildings and many houses.

The district hosts a concentration of houses that did survive the military raid, and the town hall, built in 1794, is still evocative of the architecture of that period.