He regarded Brierfield as his primary residence and returned to it when not in office and after he resigned from the United States Senate following the Secession of Mississippi in 1861.

His return to Brierfield was brief, as he was soon notified while tending the flower gardens of the house with his wife (as he recalled), that he had been elected president of the newly formed Confederate States of America and was summoned to Montgomery, Alabama, the Confederacy's first capital.

In early 1862, a small group of enslaved people liberated themselves, took control of some of the property, and fled to the Union lines near Vicksburg.

Still, fluctuating cotton prices, floods, and the cost of free (no longer enslaved) labor now denied him the income the property had once provided.

He was in residence at Brierfield in the autumn of 1889, seeing to harvest when a lingering cold developed into pneumonia, and he had to be carried onto a riverboat bound for New Orleans to receive medical attention; he died a few weeks later.