Educated at Eton College, He succeeded his great-grandfather Lord Bristol in the barony of Howard de Walden on 8 July 1803, aged four.
Charles Augustus was raised in England and does not seem to have visited Jamaica before 1846 when he took his wife, Lucy and son Frederick.
In an attempt to improve profits on his plantations, Ellis further reduced wages paid to the newly-emancipated slaves, which met with resistance and resulted in a strike.
She and her elder sister jointly inherited in 1879 the Duke of Portland's London Estate in Marylebone which had become extremely valuable.
Lady Lucy Joan Howard de Walden died in July 1899 in St. James House, West Malvern, Worcestershire .