As hat production increased, the manufacturing activity spread from Danbury to surrounding areas where the supply of fur-bearing animals such as muskrat, fox, and beaver had not yet been depleted.

[2] A small village existed at the site of Hattertown before 1821, when the Taylor and Benedict families arrived there and began hat production.

Hattertown was later to become one of the few places in the region where an entire village was directly involved in the hat trade.

There are also several secondary structures from the same time period, such as barns, sheds, and privies that contribute to the architectural significance of the area.

A schoolhouse from c.1780, now the Gregory Orchard's District School, was originally on the green (corner of Aunt Park Lane and Hattertown Road) and was moved in 1975 to its present location adjacent to the Morgan House.