As his elder brothers inherited the substantial family estates, Buller as a younger son was obliged to make his own fortune, which he achieved both from his brilliant legal career and from having married a wealthy heiress.

He has been the subject of controversy over time due to an alleged statement he made that "a husband could thrash his wife with impunity provided that he used a stick no bigger than his thumb".

On Mansfield's death, the Prime Minister William Pitt the Younger having long deliberated, passed over Buller, considered by many to be the superior lawyer,[2] and appointed Kenyon to top role.

[citation needed] In 1790 or shortly before, Buller bought the Ancient Tenement of Prince Hall on Dartmoor, Devon, from Christopher Gullet who had acquired it some ten years earlier and had already built new farm buildings on the old site.

Robert Fraser in his 1794 book General View of the Agriculture of the County of Devon (one of the General View of Agriculture county surveys series) promoted Buller's programme of fertilising the soil, improving the breeds of sheep and cattle and large scale planting of trees, as an example of best practice – even though he noted that almost all of the 40,000 larch and other conifers that Buller had planted had already died.

[11] Susanna was the niece and heiress of John Yarde (1702–1773)[12] of Churston Court, which thus became Buller's residence until he purchased Lupton House, one mile to the south.

He died during the night of 4/5 June 1800 after suffering an acute breakdown in his health during a game of piquet at his house in Bedford Square in London, shortly before had intended to resign his legal position.