Also known as Swan Pond, the house was constructed in 1797 by English architect Thomas Hope for Colonel Francis Alexander Ramsey (1764–1820), whose family operated a plantation at the site until the U.S. Civil War.

Due to their Confederate sympathies, the Ramseys fled Knoxville when the Union Army occupied the city during the Civil War,[3] and the family sold the house in 1866.

The property managed by APTA, which includes the Ramsey House, a visitor's center, and gardens, covers 101.5 acres (41.1 ha).

[2] Ramsey's rise to prominence in the failed State of Franklin (1784–1788) kept him occupied for the remainder of the decade, and it wasn't until 1792— after he had been appointed clerk for the newly formed Southwest Territory— that he decided to move to Swan Pond, which lay just outside the territory's new capital at Knoxville.

Ramsey (1797–1884),[2] who had taken an active interest in the region's history, and had built a large mansion nearby to the east, known as "Mecklenburg," overlooking the fork of both the French Broad River with the Holston River and the no longer extant Lebanon In The Fork Presbyterian Church, along the nearby and present-day Asbury Road.

[3] In the late 1850s, as one of the directors of the Bank of East Tennessee, Ramsey ran afoul of Knoxville's fiery newspaper publisher, William G. "Parson" Brownlow.

His continued attacks helped Horace Maynard defeat John Crozier Ramsey for the district's Congressional seat in 1859.

[6] At the outbreak of the Civil War, the Ramseys, who were states' rights Democrats, sided with the Confederacy,[3] drawing them further into conflict with Brownlow, who was a radical anti-secessionist.

In December 1861, John Crozier Ramsey, who had been appointed Knox County's district attorney, had Brownlow jailed on charges of conspiring to burn railroad bridges in East Tennessee.

[4] With Mecklenburg in ruins, the Ramseys, who had fled to South Carolina on approach of the Union Army in 1863, decided not to return to Knoxville.

[7] In 1927, the Bonnie Kate Chapter of the Daughters of the American Revolution placed a marker at the Ramsey House commemorating the site as the birthplace of J.G.M.

The interior has been outfitted with period furniture, most notably two Chippendale chairs given to Francis Alexander Ramsey and his wife, Peggy, as a wedding present.