The Duchess of Argyll was of the opinion that "the daughter of a private gentleman, however accomplished, was not qualified to be allied to her" even though she herself had been a mere Miss Gunning and Irish at that.

The Duke died without remarrying, even though he and Harriet Pye Esten had a daughter in 1796, Anne (1796–20 August 1844), who went on to marry Henry Westenra, 3rd Baron Rossmore.

The Duke however left the contents of Hamilton Palace to his illegitimate daughter by Mrs. Esten, Anne Douglas-Hamilton, later Lady Rossmore (died without issue).

[7] The Duke is also noted for being an early patron of the future Sir John Moore, hero of Corruna, whose parliamentary and military career was sponsored by the Hamiltons from 1779.

[8] The dance "Hamilton House" is also said to be named for the 8th Duke and his Duchess, with the changes of partner echoing the infidelities of both.