The eldest son, Alexander (1664 -1689), died of wounds while leading Frasers in support of the deposed Stuart King James II forces at the Battle of Killiecrankie.
Thomas was not able to enjoy his title and estates for long, since he was placed under sentence of death for the reckless behaviour of his son Simon in 1697.
This included Simon threatening to hang the son of a noble, and his forced marriage and rape of Hugh's widow, Amelia Murray.
As a result, both Thomas and his son were forced to flee the ancestral home of the Lovat Frasers, Dounie Castle, and take refuge in the highlands.
He was buried there, but his son Simon, who eventually became the 11th Lord Lovat, made amends for his father's exile by placing a large memorial stone for him at Wardlaw Mausoleum, near Beauly.