No primary sources authenticating its construction have been produced, only a description by Robert David Lion Gardiner in a 1976 documentary about the island.

[6][7] An earlier source that describes the settlement and early life on the island makes no mention of the shed.

He reportedly purchased the island from the local Montaukett people for "a large black dog, some powder and shot, and a few Dutch blankets.

It evidently fell under the jurisdiction of Earl of Stirling, William Alexander, who had been given Long Island by the King of England in 1636 and required Gardiner to gain his approval of the land grant through his agent James Farrett.

In 1688, when Governor Thomas Dongan granted a patent formally establishing the East Hampton municipal government, there was an attempt to annex the island, which the Gardiners successfully resisted.

[10] Gardiner's Island would remain independent of outside municipal jurisdiction until after the American Revolution, when it was formally annexed to East Hampton.

Gardiner established a plantation on the island, raising corn, wheat, fruit, tobacco, and livestock.

Privateer William Kidd stopped at the island in June 1699 while sailing to Boston to answer charges of piracy.

Indicating to Mrs. Gardiner that the box of gold was intended for the Governor of the Massachusetts Bay Colony, Lord Bellomont, Kidd gave Mrs. Gardiner a length of gold cloth,[11] captured from a Moorish ship off the coast of India,[b] and a sack of sugar in thanks for her hospitality.

The booty included gold dust, bars of silver, Spanish dollars, rubies, diamonds, candlesticks, and porringers.

A fleet of thirteen British ships sailed into the island's Cherry Harbor and began foraging and pillaging its manor house at will; they were planning to turn it into a private hunting preserve.

After the State of New York abolished slavery in 1827, many of the freed Gardiner slaves went to live in Freetown, just north of East Hampton village.

It was a 27-foot (8.2 m) square 1½ story brick building with a sixth order Fresnel Lens producing a fixed white light located 33 feet (10 m) above sea level.

A March 1888 nor'easter caused a break in the peninsula, permanently turning the point into an island (but leaving it under the jurisdiction of East Hampton).

In 1938, Gardiners Point Island was declared a National Wildlife Refuge by Franklin D. Roosevelt and transferred to the Agriculture Department.

[citation needed] During World War II, the former Fort Tyler was used for target practice and was reduced to its present state of ruin.

The state of New York briefly considered turning it into a park, but it is deemed a navigational hazard because of the possibilities of unexploded ordnance.

[citation needed] A manor house built by David Gardiner in 1774 burned to the ground in 1947, it is thought after a guest fell asleep while smoking.

Robert David Lion Gardiner and Goelet were to have a highly publicized dispute over ownership and direction of the island.

[5]In 2005, the Goelets offered to place a conservation easement on the island in exchange for a promise from the Town of East Hampton not to rezone the land, change its assessment or attempt to acquire it by condemnation.