It is named after Colonel James Swan, of Fife, Scotland, who purchased the island and some surrounding areas and organized their colonization in the eighteenth century.

[3] The town is accessible by state ferry service[4] from Bass Harbor.

The main industry is lobster fishing, which employs 40 people full-time and many others in part-time work.

[6] Swan's Island was first charted in 1606 by Samuel de Champlain's expedition, and at the time was used by local Indians as a seasonal hunting ground.

James Swan, a member of the Sons of Liberty and Boston Tea Party participant,[7] purchased the island in 1786.

In 1791, David Smith, an American Revolutionary War veteran, became the first settler of European extraction on the island.

Fish processing was the dominant industry into the 1930s, when it became no longer profitable in the area.

The state-run ferry, which began service in 1960, helped make Swan's Island a popular summer tourism destination.

A travel agency specializing in trails, describes attractions for the island that include six historic homes dating to the 1700s, a listing on the National Register of Historic Places, a lighthouse, nature features that include nesting bald eagles, waterfowl, as well as deer, and trails, hiking, and camping.