He was the second son of Stephen Adye of St Kitts and his wife Clara Payne, born 1 January 1745.
He was adopted while still young by Ralph Willett, a first cousin of his mother.
After education at Lincoln's Inn, he took to the gentlemanly life of a collector, following the example of his adoptive father Willett.
Adye was his principal heir, to plantations in St Kitts, and under a condition of the will took Willett as his surname.
[1] In later life Willett had financial troubles, and during 1813 sold collections from Merley House.