The manor lies above the River Yeo on the southern slope of the hill on top of which stand the ruins of the Anglo-Saxon hillfort called Roborough Castle.
Pilton as a borough had existed long before the Norman Conquest and was one of the most important defensive towns in Devon under the Anglo-Saxons; it is now a northern suburb of Barnstaple.
In the Book of Fees Raleigh was in the same ownership as Challacombe, which was also a Domesday manor held by Drogo from the Bishop of Coutances.
[6] This Raleigh family also held the manor of "Auvrington" (Arlington, Devon), as recorded in the Book of Fees, held from the overlord Philip de Culumbars (died 1342), of Nether Stowey, 2nd husband of Eleanor FitzMartin, sister and one of two co-heiresses of William FitzMartin (died 1326), feudal baron of Barnstaple.
In the return to the Dean Milles' Questionnaire of about 1745, the manor house of Raleigh was described as "Rawleigh in Ruins"[23] (in response to the question "Are there any gentlemen's seats and remarkable improvements in the parish?").
His son Nicholas Hooper rebuilt Raleigh House on an adjacent site slightly higher up the hill, which building survives today.
Nicholas Hooper married Mary Davie (1688–1762), eldest daughter and co-heiress of Sir William Davie, 4th Baronet (1662–1707) of Creedy in the parish of Sandford, near Crediton in Devon, by his first wife Mary Steadman (died 1690/1), daughter and heiress of ..... Steadman of Downside, Midsomer Norton, Somerset.
The heir of the Barbor family, lords of the manor of Fremington, was William Arundell Yeo, who is later recorded as a landowner at Raleigh (see below).