The family became estranged from much of Scottish society when his elder brother, David, Lord Elcho chose to back the Jacobite cause, and Bonnie Prince Charlie and, as a result of the Jacobite defeat at the Battle of Culloden in 1746 David was ineligible to inherit the estate and on the death of the 6th Earl in 1756 Francis was instead the heir.
Having done a "Grand Tour" of Europe, he chose to rebuild the old house at Amisfield in the Palladian style.
He commissioned the English architect Isaac Ware to design the house, due to his skill in the Palladian style.
[5] Charteris commissioned architect John Henderson (d.1786) in 1784 to update some interiors and add some features in the landscape.
[6] While works progressed Charteris relocated to a newly purchased estate: Gosford House on the Firth of Forth, which he enjoyed the additional pleasure of a nearby golf links.
The Edinburgh architect Stewart Kaye was involved in making a photographic record of the house prior to demolition.