Born John Luttrell, he was the second son of Simon Luttrell, 1st Earl of Carhampton by Judith Maria Lawes, daughter of Sir Nicholas Lawes, Governor of Jamaica.
He was a member of the Irish branch of the ancient family of Luttrell and a descendant of Sir Geoffrey de Luterel, who established Luttrellstown Castle, County Dublin in the early 13th century.
This was an Irish peerage and did not entitle him to an automatic seat in the House of Lords.
In 1787, out of respect after the death of his father-in-law, he assumed by Royal Licence the additional surname of 'Olmius'.
They had one child: Lord Carhampton died in March 1829, aged 89, at which time the Earldom became extinct.