He was the eldest son and heir of Sir Bourchier Wrey, 5th Baronet (c. 1683 – 1726), lord of the manor of Tawstock, a Jacobite sympathiser, by his wife (who had married him as her second husband) and first cousin Diana Rolle (born 1683), a daughter of John Rolle (died 1689), eldest son and heir of Sir John Rolle (1626–1706) of Stevenstone, near Great Torrington, Devon, Sheriff of Devon in 1682[4] and one of the largest landowners in Devon.
While living in Rome, Lady Mary Wortley Montagu recorded him as having slept with his landlady, with the encouragement of his landlord.
In 1742 he was elected to the Society of Dilettanti, a group of gentlemen who wanted to maintain an interest in the antiquarian and artistic pursuits which they had enjoyed abroad.
George Knapton (1698–1778), the official portraitist of the society, painted his portrait in 1744, in which he is depicted on board a ship holding a punch bowl inscribed with a line from Horace's Odes: "dulce est desipere in loco" (it is sweet on occasion to play the fool).
He rebuilt the pier at Ilfracombe, of which manor he was lord, and established better arrangements for English fishermen in Bremen, Hamburg, Lübeck and Copenhagen.