Frederick Hervey was born in Suffolk, and attended Westminster School before going up to Corpus Christi College, Cambridge, where he read Law.

Hervey conceived the redevelopment of the Ickworth estate to a standard appropriate to his family's wealth, with the added practical purpose of housing his extensive art collection.

[3] In 1799 the Howard de Walden barony by writ was called out of abeyance by the House of Lords Committee of Privileges in the 4th Earl of Bristol's favour.

His eldest brother, George, 2nd Earl of Bristol, became Lord Lieutenant of Ireland in 1766 and helped to secure his election as Bishop of Cloyne in 1767.

[2] Hervey was Derry's most generous philanthropist, although some of the clergy in his diocese came to regard him as cheerfully sadistic, for such instances as when he instructed any portly priests coveting plum promotions to compete in midnight runs through bogs and marshland.

[5] Given his family's wealth, Hervey both enjoyed personal luxury and expended large sums on building roads and developing agricultural enterprise within his diocese, including within the City of Derry.

[6] Buoyed by his status and popularity Bishop Hervey let slip some scurrilous talk of rebellion which led the Dublin Castle administration to contemplate his arrest.

En route at Albano, he needed overnight accommodation, which was offered by an Italian peasant by way of an outhouse; he and his wife were opposed to welcoming a Protestant into their house.

[7][8] During his European travels, and especially during his frequent visits to Rome, Hervey was known to walk around wearing a broad-brimmed white hat, many gold chains and red breeches, in what was perceived as a possible attempt to upstage his Catholic rivals.

[10] When Hervey died in 1803 in Lazio, Italy, his son Frederick succeeded as 5th Earl of Bristol as well as to the family estates including Ickworth House.