He built himself a relatively modest mansion in the conventional style of the day, which can be seen in Canaletto's painting of Whitehall.
[1] In the late 1850s, the 2nd Duke of Montagu's descendant, Walter Montagu Douglas Scott, 5th Duke of Buccleuch, one of the United Kingdom's three or four richest landowners, replaced the Georgian house with one of the grandest private mansions in London.
It was designed by the versatile Scottish architect William Burn in the style of a French Renaissance chateau.
The interior featured a top-lit central saloon and a grand staircase, heavily coffered ceilings and elaborately carved furnishings.
The site forms roughly the southern half of that of the current main Ministry of Defence building in Whitehall.