It had been built on a large tract of land acquired by Captain Francis Eppes in 1635 and by the time of the American Civil War it was the center of a plantation covering more than 2,300 acres (9.3 km2).

When the war came to Petersburg two years later Mrs. Eppes and the children fled again, this time to her mother's home in Philadelphia, Pennsylvania.

During the last years of the Civil War, from the port there, the City Point Railroad was used to support the Union forces during the Siege of Petersburg 1864–65.

The successful capture of Petersburg and its network of railroads was the key to the fall of the Confederate capital city of Richmond, ending the war less than a week later.

Not until March 1866 with the last Union regiments gone and the property back in his name did his wife and children return home to pick up the pieces and start anew.