[1] Fermor was the second but eldest surviving son of Sir William Fermor, 1st Baronet (1621-1661) (alias Farmer), of Easton Neston, Northamptonshire, by his wife Mary Perry, widow of Henry Noel, second son of Edward Noel, 2nd Viscount Campden and a daughter of Hugh Perry of London.

[2] Leominster re-built the mansion house at Easton Neston and planned the gardens and plantations, the wings being to the design of Sir Christopher Wren with the house completed 20 years later in 1702 to the design of Nicholas Hawksmoor.

He adorned the whole with part of the Arundel marbles which he had purchased and which his son attempted to restore with the assistance of the Italian sculptor Giovanni Battista Guelfi, a scholar of Camillo Rusconi.

Horace Walpole wrote the following to George Montagu on 20 May 1736, "Coming back, we saw Easton Neston, where in an old greenhouse is a wonderful fine statue of Tully haranguing a numerous assembly of decayed emperors, vestal virgins with new noses, Colossus's, Venus's, headless carcases, and carcaseless heads, pieces of tombs, and hieroglyphics."

A description of Easton Neston and its art treasures is included in the Catalogue of the Duke of Buckingham's Pictures.