In September 1608 he argued with the Earl of Pembroke over a game of cards.
On the next day while hunting, Wharton hit Pembroke's page with a stick.
[2] On 8 November 1609, another argument over playing cards escalated between Wharton and Lord Blantyre's son Sir James Stewart, Master of Blantyre, and husband of Dorothy Hastings.
[4][1] The Venetian ambassador Marc' Antonio Correr wrote that King James moved out of London in response to the duel, to avoid any bad feeling against his Scottish courtiers.
He identified Wharton as a brother of the sister-in-law of Henry Wotton, the English ambassador in Venice.