Upon the accession of the first Hanoverian monarch George I and his proscription of the Tories in 1715, Barrymore was forced to sell his regiment.

In 1740 he conspired with English Tories for a Stuart restoration aided by a French invasion and visited Cardinal Fleury to persuade him to support it.

[1] In 1743 Louis XV of France's master of horse, James Butler, 2nd Duke of Ormonde, travelled to London to meet Barrymore and other Tory peers to conspire to French invasion.

The group also sent a communication to the French king through the Jacobite agent Francis Sempill, requesting support for a Frenco-Jacobite invasion.

In February 1744 the British government discovered from a spy in their service in France the English members of the conspiracy and Barrymore was arrested.