In 1747, he married Margaret Luttrell, the heiress of Dunster Castle and added his wife's name to his own to comply with her late father's will.

The newly named Fownes Luttrell altered the castle considerably, remodelling its interior and park and building the Conygar Tower, a folly in the grounds.

[16][n 1] Her father left debts due to his lifestyle and also provisions in his parents' wills that forced him to support his niece, Anne Luttrell, a daughter of Francis, his brother.

[18] To improve his wife's estate, Fownes Luttrell obtained a sanction from the Court of Chancery to sell the manors of Heathfield and Kilton but found no buyers;[19] he subsequently attempted to sell the manor of Minehead, and also offered at least one of the Parliamentary seats for the Borough of Minehead for £30,000, but there were no serious bidders, largely due to the high prices and the fact that Fownes Luttrell had in fact not inherited much control over the seats he was putting up for sale.

[2][20] However, frugal spending helped to recover the estate and Fownes Luttrell invested in 1760 in the manor of Foremarsh, near Dunster, which he purchased from John Poyntz.

[21] Beginning in 1747, Fownes Luttrell made numerous and significant alterations to Dunster Castle, redecorating and redesigning it in the Rococo style.

[27][28][29] In 1764, the Lower Ward of the Castle was reduced to one level, while several adjoining walls were removed and the road passing through the gateway was covered over.

[20] An intervention from the Duke of Newcastle, at Egremont's request, led to Boone and Whitworth forming an alliance against Fownes Luttrell.

[20] During this Parliament, Fownes Luttrell entertained voters at Dunster Castle and in the village, where he laid on expensive feasts.

[2] According to Maxwell Lyte, "it does not appear that [he] had any real zest for Parliamentary life [and was] probably far happier with his hounds and his fighting cocks in Devon or Somerset.

"[37] However, it appears that the Prime Minister, Lord North, believed that in return for the Government's support in 1768, Fownes Luttrell had promised to return the Government's candidate, Thomas Pownall, at the next election; when that time came, in 1774, North supported Fownes Luttrell and "warned off" his rival, Charles Whitworth.