The buildings within the district primarily represent the period of Edgartown's growth in the 19th century, featuring the elaborate houses of wealthy ship captains, as well as significant public buildings such as the Dukes County Courthouse and Jail, the Federated Church, and the Whaling Church.

[1] Edgartown was settled after the area was granted in 1641 to Thomas Mayhew, having previously been occupied by the Wampanoag people, and was incorporated in 1671, when Martha's Vineyard was part of New York.

Only portions of First Period construction have survived in Edgartown; one example is the Captain D. Fisher House on North Water Street (c. 1704).

However, side-hall plans that were typical of this time in other parts of Massachusetts were uncommon here, Most of the buildings from this period were cottages of one and a half stories, with modest styling such as corner pilasters.

Because of the decline in the whaling industry and the village economy, there are comparatively few Victorian period buildings.