The titles of Baron Annesley, of Castlewellan in the County of Down, and Viscount Glerawly, in the County of Fermanagh, were created in the Peerage of Ireland on 20 September 1758 and 14 November 1766 respectively for his father William Annesley, who sat as Member of the Irish Parliament for Midleton.

He had earlier represented seven different constituencies in the Irish Parliament and served as a Commissioner of Customs for Ireland.

His eldest son, the third Earl,[4] sat in the British House of Commons as the representative for Downpatrick.

He sat as Conservative Member of Parliament for Great Grimsby and was an Irish representative peer in the House of Lords from 1857 to 1874.

His line of the family failed on the death of his only son, the sixth Earl, who was killed during the First World War.

During his life in 1884 the Earl had a property named the 'Moorish tower' constructed, only for it to become ruin before a century later.