He was the eldest son and heir of Sir Edward Bampfield (died 1528) of Poltimore by his wife Elizabeth Wadham, the widow of John Warre of Chipleigh (see Chipley Park, Somerset), second son of Sir Richard Warre of Hestercombe, and a daughter of Sir Nicholas Wadham (died 1542) of Merryfield, Ilton in Somerset and of Edge, Branscombe in Devon.

Bampfield's father died when he was two years of age, and the 18th-century genealogist Thomas Wotton related this tradition about his childhood:[2][a] ...he became a prey to some great person, who carried him into a distant country, and bred him up in the drudgery of the family, concealing from him his quality and estate, and at last made him his huntsman; but one of his tenants, (being his nurse's husband,) discovering where he was detained, made him acquainted with his fortune; the truth of which he convinced him of, by a remarkable mole which he had in his back, and brought him away privately to Brimpton (the seat of John Sydenham, Esq; who assisted him in his return to Poltimore, and soon after gave him his daughter in marriage.)

In confirmation of which, he lieth at length with a hound at his feet, under a monument in Poltimore church...However, the editor of the 1771 edition of Wotton's genealogy added "Having received no account from the family, concerning this particular, I do not presume to give it as authentic.

Her niece Elizabeth Sydenham was the wife of Admiral Sir Francis Drake (c. 1540 – 1596).

It comprises two recumbent stone effigies, of Richard Bampfield and his wife, under a low canopy supported by arched openings and columns.