In 1624, a ship with 30 Protestant Walloons (people from what is today southern Belgium) landed in New Netherland; 18 of the men were sent to the location near present-day Albany.

The Dutch party was ambushed and three men were killed approximately a mile from the fort, roughly where Lincoln Park and Delaware Avenue are sited today.

[citation needed] When the Dutch established the Charter of Privileges and Exemptions in 1629 setting up the patroon system, Kiliaen van Rensselaer established his patroonship of Rensselaerswyck, surrounding Fort Orange on 24 miles (39 km) of shoreline along the Hudson River and 24 miles (39 km) inland on each side of the fort.

[citation needed] In 1630, Gillis Hoosett purchased in van Rensselaer's name the lands to the south and north of the fort from the natives.

In 1634 the commander of Fort Orange ordered Harmen Meyndertsz van den Bogaert west into the Mohawk Valley and Indian country, for the purpose of understanding why the fur trade had declined.

In response, the patroon's agent, Commander van Schlechtenhorst, decided to expand settlement to "within pistol shot of Fort Orange".

In 1652, Stuyvesant, to settle this dispute once and for all, set up a "Court of Justice for the Village of Beverwyck and its dependencies", the first municipal government for the future city of Albany.

[citation needed] On September 8, 1664, the English, after sending numerous war ships to New Amsterdam, demanded the surrender of New Netherland and came to terms with the Dutch.

Johannes De Decker sailed on that day from New Amsterdam to Fort Orange to rally the troops and settlers to resist English rule.

[citation needed] It was not until September 24, 1664, that vice-director of New Netherland Johannes de Montagne surrendered the fort to the English, and Colonel George Cartwright took command.

[7] In 1886, as part of the bicentennial of Albany's incorporating document, the Dongan Charter, the city erected a bronze metal engraved tablet at the site of the northeastern bastion of Fort Orange.

In the 1930s the tablet was moved during construction of the first of two Dunn Memorial Bridges (named for Private Parker F. Dunn (1890–1918, a local veteran / soldier of the United States Army, killed on the Western Front in northern France, during the First World War (1914/1917–1918), and awarded the congressional Medal of Honor, the current second / replacement span built in 1969.

The Albany Institute of History and Art has a cannonball labeled as "Dug up at Fort Orange site July 22nd 1886", the date / year that the bicentennial marker was placed.

The tablet was moved again in 1971 after the 1970–71 archeologist historical excavations discovered remnants of the Dutch fort which were prompted during the construction project for nearby larger / modern Interstate 787 and the highway interchange with the South Mall Expressway.

The greatest number of fish bones and scales were found in a pit 20 feet (6.1 m) south of the pebbled entrance path dating from before 1648.

[17][18] The site was in continual use and artifacts recovered include 1830s creamware or pearlware and 1860s copper bottom coffee pot remnants.