He was described by Horace Walpole, in his Memories, as "the jolly old Archbishop of York, who had all the manners of a man of quality, though he had been a buccaneer, and was a clergyman; but he retained nothing of his first profession, except his seraglio".

A popular story recounts that he spent these years sailing with buccaneers, either as their chaplain or as a pirate himself; there is little evidence either way, although a record of 1681 notes that he was paid £20 (equivalent to £3,960 in 2023)[3] by Charles II for "secret services".

The Dictionary of National Biography mentions "his reputation for carnality" and "the laxity of his moral precepts", while Brewer's Rogues, Villains and Eccentrics comments that "[his] behaviour was seldom of a standard to be expected of an archbishop.

He was famously ejected by John Disney, the vicar of St Mary's Church, Nottingham, after a confirmation service during which he asked for his pipe, tobacco and ale.

In a 1780 letter to David Dalrymple, Horace Walpole gave a lengthy description of Blackburne: He was perfectly a fine, gentleman to the last, to eighty-four; his favourite author was Waller, whom he frequently quoted...

On the King's last journey to Hanover, before Lady Yarmouth came over, the Archbishop being With her Majesty, said to her, "Madam, I have been with your minister Walpole, and he tells me that you are a wise woman, and do not mind your husband's having a mistress."