Early growth was due in part to industry that developed on the Blackberry River near the town center, of which no significant traces remain.

The major element of the town center, the former Stoeckel estate, is now home to the Yale Music School's summer programs.

[2] The historic district covers about 90 acres (36 ha), most of which consists of the Stoeckel estate, located west of the town green.

The triangular town green is the district's main focal point, with two churches (one early 19th century Federal style, the other later Romanesque) facing it from the west, the Norfolk Public Library (a handsome Shingle style building designed by George Keller in 1888) facing it from the north, and mainly older homes facing it from the east.

Several buildings in this area were designed by the noted New York architect Alfredo S.G. Taylor, who summered in Norfolk.