In 1799 he assumed by royal licence the additional surname of Sidney on succeeding to the estates, including Penshurst Place in Kent, of his maternal grandmother.

In 1835, fourteen years before succeeding his father in the baronetcy, he was raised to the Peerage of the United Kingdom as Baron De L'Isle and Dudley, of Penshurst in the County of Kent.

His grandson, the fifth Baron (who only held the titles for two months in 1945 after succeeding his elder brother), notably served as Mayor of Chelsea and was a member of the London County Council.

In 1956, he was created Viscount De L'Isle, of Penshurst in the County of Kent, in the Peerage of the United Kingdom.

Close to it is the parish church of St John the Baptist, where the Sidney Chapel houses many memorials to the family.