The Ellis family owed their wealth to sugar plantations at Montpelier in the Colony of Jamaica, although Charles lived in England and does not seem to have visited there until in his mid-forties.
The family's English properties comprised a house in Audley Square, London, and an estate at Seaford in Sussex.
[4] The young Lord Howard de Walden (1799–1868), aged four when he inherited the title from his great-grandfather, eventually married Lady Lucy Cavendish-Scott-Bentinck, youngest daughter of William Bentinck, 4th Duke of Portland, and sister and co-heiress of the reclusive John Bentinck, 5th Duke of Portland.
Building leases granted from the mid-18th century began to make huge financial returns from the 1870s and quickly made the Howard de Walden family one of the wealthiest in the country.
Aged 46 he married Blanche Holden, a beauty 25 years younger than he, whom he subsequently divorced, to the scandal of the day.
[7] He adopted the name of Scott-Ellis instead of Ellis and was a keen sportsman and playwright with a particular interest in promoting the Welsh performing arts.
By royal warrant dated 25 June 2004, the Queen called the barony of Howard de Walden out of abeyance in favour of the eldest daughter, Hazel Czernin (born 1935).