James Morgan and his brother William also operated several nearby grist and sawmills, as well as a business several counties to the west in Lynchburg.

There, Dr. Loughborough owned a sugar plantation he called "Esperance" and after providing medical service during an epidemic in New Orleans which earned him a civic commendation, was elected to the Louisiana state senate.

[4][5] Dr. Loughborough had been born near Washington, D.C. where his father Nathan Loughborough served as a Treasury Department official under President John Adams and had become wealthy with investments in a bank as well as the Rockville Turnpike and later the Chesapeake and Ohio Canal,[6][7] and had received his medical education in Philadelphia.

During the American Civil War, Dr. Loughborough aligned with the Confederacy, and returned to Virginia after his son Nathan died at the Battle of Cheat Mountain.

A six-bay-wide, two bay-deep rear addition designed by noted English architect Edmund George Lind (1829–1909) was added in 1859, creating a "T"-plan dwelling.