It was the home of Robert Toombs (1810–85), a U.S. representative and U.S. senator from Georgia who originally opposed Southern secession but later became a Confederate Cabinet official and then a Confederate general during the American Civil War.

The interior of the house has been decorated to reflect its mid-19th century occupation by Robert Toombs, and includes his private library.

Toombs is credited with a number of alterations, including the front colonnade and the west wing.

[3] Robert Toombs was one of the American South's prominent pre-Civil War orators.

He fled the country at the end of the war, eventually returning to a successful law practice.