In May 1695, Lord Elgin was accused, almost certainly with good reason, of having conspired to plan the restoration of King James II and in February 1696 he was imprisoned in the Tower of London,[4] but admitted to bail a year later and allowed to leave England for Brussels.

Some historians have accused him of double-dealing in swearing allegiance to William III while plotting the restoration of James; others argue that his true loyalty was to the institution of the monarchy, and that he supported whichever monarch seemed best fitted to rule at any given time.

[5] William III clearly did not regard him as a dangerous character, as shown by the fact that he was left in peace once he fled from England; he was fortunate in having a great many friends and very few enemies.

Though he changed allegiance himself he had no patience with time-servers: he detested Sunderland (while admitting that he was good company)[7] and in 1689 told his cousin Danby that for his treachery to James II he deserved to "be knocked on the head".

Historians have praised them highly, particularly for the vivid portraits of the leading figures in British life, including James II, William III, Danby, Sunderland, Lauderdale and Halifax.