In the mid-1860s, Lord Winchilsea experienced serious financial difficulties due to his gambling addiction, which eventually forced him to leave his property at Eastwell Park in Kent.

Lord Winchilsea had been obliged to vacate the property some time prior to December 1868,[1] and he was formally adjudged bankrupt on 5 October 1870.

His half-brother, Henry, later the 13th Earl of Winchilsea, wrote about him in his diary despite what happened with the family fortune "I hope he knew how fond of him we all were.

Lord Winchilsea, his son and his second wife are buried in the cemetery of the now ruined church of St Mary the Virgin at Eastwell Park.

Through his daughter, Lady Constance, he was the grandfather of Gordon Howard, 5th Earl of Effingham (1873–1946), who married Rosamond Margaret Hudson and later, Madeleine Foshay, and Capt.