Charles Island is accessible from shore via a tombolo (locally referred to as a sandbar), which is exposed at low tide.

However, Milford locals note that earlier in the 20th century the window was much greater, but erosion has reduced the size and width of the island and sandbar.

Kayakers paddling on the side of the island not facing the Connecticut shore can be surprised by the wind and rougher waves, according to a spokesman for the Milford Fire Department.

Deal made a failed attempt to raise tobacco on the island – one of the first such efforts in Connecticut.

After the Civil War the resort closed and the island was leased to the George Miles Company as a base for their fish fertilizer operations.

The waters west of the island have been a site for oyster farming[5][6] since 2013, in an area of approximately 3.4 acres (1.4 ha).