A viscountcy was to be conferred on his father in 1815, but he died before the patent had passed the Great Seal and the title was never given to his son.
[1] In his youth, Gardner was a member of the literary salon established by the Countess of Blessington and the Count D'Orsay.
Lord Gardner first married Frances Margaret Hughes (12 October 1814 – 3 December 1847) in 1835.
They had several children born before and after their marriage; as his former mistress, she was unacceptable in Victorian high society.
His son Herbert Gardner, born two years before his parents' wedding, became a Liberal politician and was created Baron Burghclere in 1895.