He had an elder brother, Thomas (d.1559) and a sister, Margaret, who married Sir Matthew Arundell (c. 1533 – 24 December 1598) of Wardour Castle.

However, only a few months later, on 27 August 1549, Sir Henry Willoughby was slain on Mousehold Heath in the suppression of Kett's rebellion.

George Medley was arrested, and imprisoned in the Tower for a time, and when he was released in May 1554 was unable any longer to care for Francis and Margaret.

[5] In August 1559, Thomas Willoughby died from 'overheating' while out hunting, leaving no issue, and Francis inherited the family estates.

He then decided to make his distant relative Percival Willoughby of Bore Place in Kent his principal heir, if he had no son, through marriage to his eldest daughter Bridget.

[9] Willoughby entered into a number of speculative ventures, including growing and processing woad at Wollaton, and a plantation in Ireland.