The chapel is located at the western end of the house and was used for Roman Catholic services when the Giffards lived there in the late 16th century following the Dissolution of the Monasteries.
The senior branch of the Giffard family was seated at nearby Brightley in the adjoining parish of Chittlehampton.
A document[citation needed] states that Accott was held by Richard Cotell from a feudal overlord named Drogo de Lington in 1127 after his ancestors were granted the estate by a Bishop of Exeter.
It was then sold by three sisters and joint-heiresses who had inherited the estate following the death of their two brothers who killed each other in a quarrel.
For a further 300 years the prominent Chichester family of Hall (in the same parish) owned the farm up until the early 20th century.