The Eastville community, a mixed diaspora of Irish, Native American and African-American workers, was established during the 1830s , coinciding with the boom times of Sag Harbor whaling.

In 1857, trustees Elymus Derby, Samuel Butler and David Hempstead of the AME Zion church purchased the land for a cemetery from Hannah and Anna Maria Solomon.

[citation needed] Beginning in the early 1800s until the mid 1900s, the portion of the historic Sag Harbor district known as Eastville was home to a multi-ethnic population of free Blacks, European immigrants and Native Americans.

[5] Today Eastville retains its ethnic diversity, with some flight of African-Americans and influx of new residential homebuilders, a change that threatens the summer cottages built during the 1950s for a overbuilt esthetic of mega-mansions more suited for the economies of scale to match the Hamptons chic.

[13] Since 1985 the society has kept the preservation of the Eastville community foremost, and was also instrumental in getting recognition of the adjacent Sag Harbor Hills, Azurest, and Ninevah Beach Subdivisions Historic District designation by the National Park Service.