[1] Hendrick Christiaensen chose Castle Island to build Fort Nassau, in 1614 or 1615 as a dual warehouse and military defense structure and named the fort in honor of the stadtholder of the United Netherlands, who was of the House of Orange-Nassau.

Nineteenth and early-twentieth century historians claimed that around 1540, French fur traders built a stone "castle" or fortified trading post on Castle Island in the location where Fort Nassau was later built.

[5] In 1617 a freshet damaged the fort to such an extent that it was abandoned and rebuilt on more secure ground at the mouth of the Normans Kill (called the Tawasentha by the natives) with the Hudson River.

[5] In 1624, the Dutch built Fort Orange about a mile to the north, at current Albany.

The island was part of the town of Bethlehem until 1926 when it was annexed to the city of Albany.