Quest-Ritson was born on 20 June 1947 in Portsmouth, the eldest son of Gerald Ritson, a lieutenant in the Royal Marines, and Margaret Quest Thomson.
He is the grandson of the mining engineer Professor John Ritson and the great-great-grandson of Thomas Alexander Dickson, the Liberal and Irish Nationalist MP.
[citation needed] He attended Stoke House preparatory school in Seaford, Sussex and won gardening prizes there from the age of nine.
He was privately educated at Winchester College, where his housemaster was Dr Peter Partner, the historian of late-mediaeval Italy.
His first published horticultural book was The English Garden Abroad (1992) and he has written more than ten further guides over the following thirty years.