Castle Williams is a circular fortification of red sandstone on the northwest point of Governors Island, part of a system of forts designed and constructed in the early 19th century to protect New York City from naval attack.

Each of its four levels had 13 casemates (or pairs of barbette mounts on the fort's roof) that could hold 26 cannons of varied caliber.

[3] Before its completion, Colonel Henry Burbeck, commanding the defenses of New York City, issued an Army order on November 24, 1810, that named the castle for its designer and builder: "In future the Stone Tower on this Island (by the approbation of the Secretary of War) will bear the name of CASTLE WILLIAMS, in honor of the commandant of the United States Corps of Engineers, who designed and erected it."

Improvements that included the installation of central heating and plumbing were most likely made in 1895 when Castle Williams was designated one of the U.S. Army's ten military prisons.

Concrete floors and brick partitions were installed in the casemates of the second tier, and steel security sashes and doors replaced those made of wood.

Instead it became a community center that provided a nursery, meeting rooms for scouts and clubs, a woodworking shop, art studios, a photography laboratory, and a museum.

With the relocation of those civilian functions to new locations on the island in the mid-1970s, the castle ended its military career in a state of mild neglect as a storage facility and landscape shop for the Coast Guard.