The Astor area and much of the land along the St. Johns River was inhabited by Timucua natives prior to settlement by Europeans.

Early attempts at settlement included an English trading post in 1763, and in 1822 a plantation growing sugar cane and oranges was established by Jewish immigrant Moses Elias Levy.

By 1838 the Seminole Wars had begun and the United States government established Fort Butler to defend the river as the primary route of transportation inside Florida.

These earliest efforts at settlement all met with failure due to war or disease, and until the 1870s the area was largely deserted.

New settlers arrived by steamboat to the town which Astor had endowed with a church, schoolhouse, botanical garden, and free cemetery.

Over the next twenty years Astor saw his town grow, but the Manhattan name never caught on.

[9]: 23–28 John Jacob Astor IV inherited his father William's estate and continued to promote the town and their business interests in Florida.

[9]: 33, 35 Today the community of Astor is largely reliant upon tourism, and is a popular spot for winter visitors from the north and for fishing, hunting, and boating enthusiasts.