The Greek Revival house was built in 1852 by Robert Woolfolk on behalf of Lloyd Tilghman, who moved with his family to Paducah that year.

Tilghman was a United States Military Academy graduate, having finished 46th out of 49 in his class, but spent less than a year as a Second Lieutenant.

Woolfolk was pro-slavery and flew a Confederate flag in response, sparking a riot in December 1861 that included U.S. soldiers, particularly those of the 11th Indiana Regiment.

The incident would begin U.S. Brigadier General Charles Ferguson Smith's decline as he saw his subordinate, Ulysses S. Grant, raised above him almost immediately.

[6] On December 1, 2008, the Sons of Confederate Veterans purchased the home from the foundation, with each group paying half of the remaining $150,000 mortgage.