They had purchased the land in the summer of 1640 from the group of Indians related to the Pequot of New England, who lived in the territory they called Corchaug (now Cutchogue).

In most histories Southold is reported as the first English settlement on Long Island in the future New York State.

The Dutch colony was the western part of Long Island, and the English dominated the east.

The harbor at Greenport, on the North Fork, became important in trade, fishing, and whaling, because it rarely froze over.

When the Dutch took control of the colony of New York in 1673, the English-settled eastern towns, including Southold, East Hampton, and Southampton, refused to submit; the Dutch attempted to force the matter by arms, and the colonists of the towns repelled them, with assistance from Connecticut.

When New York was retaken by the English in 1674, these eastern towns preferred to stay part of Connecticut.

Governor Sir Edmund Andros threatened to eliminate the residents' rights to land if they did not yield, which they did by 1676.

Hobart's brother Josiah was one of the earliest settlers and initial trustees of East Hampton, Long Island, as well as High Sheriff of Suffolk County.

[8] The name Southold is believed to be an elision of Southwold, a coastal town in the corresponding English county of Suffolk.

John Youngs, the minister who was one of the founders of the Town, was born and brought up in Southwold, England.

The Town's name also may refer to a "holding" to the south [of New Haven]), from whence the original settlers hailed.

In the late 19th century, the Long Island Rail Road extended its line on the North Shore to Greenport.

Due to the light on the North Fork from water on both sides, the area attracted many artists, including William Merritt Chase.

Robins Island, a protected open space in Great Peconic Bay, is also part of the Town of Southold.

A handful of world-renowned artists have lived in the town including; Douglas Moore,[15] Robert Berks, and Walt Whitman.

Local music venue and restaurant, The Green Hill Kitchen, has seen performances from modern jazz greats such as Tommy Campbell (musician), Gil Goldstein, Alex Sipiagin, and Morris Goldberg.