[2] In 1190, Frethesant Paganel married Sir Geoffrey de Luterel, and the estate passed into the Luttrell family, whose arms appear above the gateway of the house.

The estate remained with the Luttrells until 1406, then passed through various owner before coming into the possession of the Crown during Richard III's reign.

[3] Hutton's descendant Col Robert Byerley, of Byerley Turk fame, began negotiations around 1681 with Sir Patience Warde, Lord Mayor of London, to purchase the estate;[1] the purchase was eventually completed in 1703 by his nephew, also called Patience Warde.

[2][5][4] During the First World War, the house was opened as Hooton Pagnell Auxiliary Military Hospital, with Warde-Aldam as commandant and matron.

Mary Betty Norbury received the Hooton Pagnell estate as a gift from her father, Col William St Andrew Warde-Aldam, in 1952, and she and her husband assumed the surname of Warde-Norbury by Royal Licence in 1958.