He appears to have been a descendant of General Roger Handasyd an early governor of Jamaica.
His more ancient ancestry can be traced back to Edgar of Wedderlie, who held lands from the Earls of Dunbar, during the reign of Malcolm III of Scotland.
He moved to Jamaica with his brother in the late 18th century to run the Wedderlie Estate (named after his family home and probably established by their father), and the Osborne Estate (created around 100 years earlier by a Dr Osborne).
He appears to have already had kinsfolk in Jamaica including several half-brothers and sisters fathered by Alexander Edgar with negro slaves.
A dispute over his will, between his brother Alexander Edgar and Sir Simon Haughton Clarke was heard by the Court of Session in Edinburgh under Allan Maconochie, Lord Meadowbank in 1810.