Known as 'The Fox', Lovat became deeply involved in the Jacobite cause, which aimed to restore the deposed House of Stuart to the thrones of Scotland and England.
He attempted to recruit Scottish nobles to the cause, carrying messages to Scotland, but his dealings led to ten years imprisonment in France.
Adam's last work, the project only progressed to the supply of stonework to the site: construction never started since the Jacobite Rising of 1745 intervened.
His son Simon Fraser, 13th Lord Lovat (1828–1887), built the present Beaufort Castle, to designs by James Maitland Wardrop, incorporating part of the 18th-century house.
The castle was sold in 1994 to Stagecoach director Ann Gloag by the 15th Lord Lovat, to meet inheritance taxes.
[5] The remains of Dounie Castle stand beside the house, and consist of a single wall, 11 metres (36 ft) long and 1.5 metres (4 ft 11 in) high, with a plaque stating that it is "the ruin of Castle Downie, the ancient stronghold of the Frasers of Lovat, built c. 1400, and destroyed by Cumberland after the battle of Culloden".