Vernon was the son of Robert Percy Smith, of 20 Savile Row, London, and of Cheam, Surrey, and the nephew of The Rev.

The Whigs fell from office in November of that year, but returned already in April 1835, when Vernon was appointed Secretary of the Board of Control by Melbourne, which he remained until 1839.

He did not hold office again until February 1852, when he was made Secretary at War in the first administration of Lord John Russell.

When the Liberals (as the Whigs were now known) returned to office in 1855 under Lord Palmerston, Vernon was appointed President of the Board of Control, with a seat in the cabinet, a post he retained until the government fell in March 1858.

Lord Lyveden died in November 1873, aged 73, and was succeeded in the barony by his son Fitzpatrick.