The mansion is located at 300 West Chulahoma Avenue in Holly Springs, a small town in Marshall County, Northern Mississippi.

[1] It was built for Colonel Harvey Washington Walter,[3] a lawyer who served as the President of the Mississippi Central Railroad.

[6][7] During the American Civil War of 1861–1865, Walter, who was opposed to secession, invited Union General Ulysses Grant to live in the house.

[2][3][6] As Confederate General Earl Van Dorn liberated Holly Springs, he was not permitted to enter the house until Julia Grant had gone outside.

[8] Other sources, including Julia Grant's memoirs, give a slightly different version of events saying that she had left Holly Springs before Van Dorn and his troops arrived.

Julia Grant's carriage was burned and her horses taken, but this would have been in line with Van Dorn's plan to remove or destroy anything in the area that may have benefited the Union army.

The Lynns fulfilled the ambition of former owner Oscar Johnson by adding a walking park with water features and botanical gardens on an adjacent piece of property.