At the time of European encounter, it was occupied by the Manhanset tribe, an Algonquian-speaking people related to the Pequot and other Algonquians of New England.

"[4][5] Shelter Island was included in the original Plymouth Company land grant made by James I of England in 1620.

Alexander gave James Farret power to act as his agent and attorney in colonizing Long Island.

Farret in turn sold the islands to Stephen Goodyear, one of the founders of the New Haven Colony.

On March 23, 1652, he made the purchase official by agreement with Youghco (called Poggatticut), the sachem of the Manhanset tribe.

The other owners, Sylvester’s brother Constant, and Thomas Middleton, never came to Long Island.

[9] In 1652 Sylvester constructed a house on the island for his 17-year-old bride, Grissel (also spelled Grizzel)[9] Brinley from London.

It raised food crops, as well as livestock for slaughter, sending casks of preserved meats and other supplies to Barbados.

Sylvester and his associates were part of the Triangle Trade between the American colonies (including the Caribbean), Africa and England.

In 1695, William Nicoll, a resident of Islip, bought from Giles the area now called Mashomack Nature Preserve.

Over time these estates and parcels were split and divided by marriage and purchase, so that by the early 18th century, 20 families lived on Shelter Island.

[8] James Nicoll Havens, a member of the New York Provincial Congress, built a home on the island in 1743.

Jonathan Nicoll Havens (1757–1799), born on Shelter Island, was a member of the First Continental Congress in 1774.

Shelter Island Heights was developed in 1871 as a summer resort for camp meetings, after a group of Brooklyn businessmen purchased the Frederick Chase estate for this purpose.

It was developed by the Shelter Island Grove and Camp Meeting Association of the Methodist Episcopal Church.

During the eight years from 1872 to 1880, when camp meetings were held here, about 70 summer cottages were built in the Heights.

White Hill is located above the North Ferry landing in Shelter Island Heights.

The remains of his estate are reinforced concrete retaining walls and a "Japanese" footbridge from the garden, built by Ernest L. Ransome about 1898, behind Merkle Lane.

Another 19th-century millionaire who had an estate on Shelter Island was Artemas Ward (1848–1925), a pioneer in mass-market advertising.

Ward made millions of dollars by monopolizing all advertising on New York City elevated trains, subways, and streetcars.

After years of being tied up in court, the vestiges of Ward’s original estate in the Shorewood section of the Island (including a manor house, formal Italian gardens, and a concrete water tower) have been renovated.

On West Neck Harbor, developers Albert and Fred Dickerson built houses for what is today called Montclair Colony.

Homes were built on Silver Beach, Ram Island, and Hilo Farms.

It was not until after World War II that summer residents started returning in larger numbers.

Nearly one-third of the island is owned by The Nature Conservancy to be preserved in a wild state.

In the 1996 presidential election, it was the only town on Long Island to vote for Bob Dole.