He also established a small village and arranged for plantations of Douglas firs near the mansion house.Ferguson was born at Pitfour in 1700 shortly after his father, also named James, had purchased the estate.

The eldest, James (1736–1820), became a politician; Patrick (1744–1780) invented the Ferguson rifle, a breech-loading flintlock weapon;[3] and the youngest, George (1748–1820), became Lieutenant Governor of Tobago in 1779.

[1] In his memoirs, John Ramsay of Ochtertyre described the first Laird was as "having been an adventurer in the South Sea" and that he "would have been a ruined man, but for his son's exertions.

"[5] Wilson-Smith interprets this to mean the first Laird had lost a great deal of money investing in the South Sea Company, but the expansive lands he procured generated sufficient revenue to continue payments on the debts until the situation was later fully remedied via the financial acumen of his son, Lord Pitfour.

[1] He was appointed as Vice-Dean of the Faculty of Advocates in January 1759 and became Dean when Robert Dundas was elevated to Lord President of the Court of Session in June 1760.

[7] In 1764, the Earl of Mansfield successfully petitioned George III to allow Ferguson to be promoted to fill the vacant position of judge sitting in the Court of Session.

[12] The records of the Spalding Club include a soldier's letter which indicates it is likely Ferguson was willing to offer refuge to Jacobites who participated in the Battle of Culloden.

[5][10][13][14] The 1773 papers of George III contain details of Pitfour accompanying Lord Justice Clerk Thomas Miller on the Northern Circuit from the end of April until 20 May.

[16] This was endorsed by the diarist John Ramsay of Ochtertyre (1736–1814), who remarked that Pitfour was one of a small number of barristers who he found satisfying to hear speak.

His desire to be fair and reach compromise had stood him in good stead as an advocate, but as a judge, these characteristics drew criticism from his contemporaries.

Writing in 2008, local historian Alex Buchan speculated that Pitfour's hard life travelling around Scotland took too much of a toll on him, leading to a rapid decline in his health and ability.

[7] After the Earl Marischal returned to favour and the York Buildings Company suffered financial difficulties, he bought the land back for £31,000 at an auction, to the delight of friends and others present.

The land, adjacent to the Pitfour property, incorporated St Fergus and Inverugie Castle, the former seat of the Earls Marischal.

[25] The addition of this extensive property made Pitfour the largest estate in the area, stretching over 30,000 acres (120 km2) from Buchanhaven in Peterhead and along the course of the River Ugie to Maud.

Douglas firs were planted near Saplinbrae and Deer Abbey from seed sent from Canada by Pitfour's brother-in-law, General James Murray.

[28] Fitted with a heavy freestone slab roof to hinder bodysnatchers, it bears the simple inscription: "Jacobus Ferguson de Pitfour.