[2] Its toponym is derived from the Old English Aldeberie, meaning "old fortified place",[3] suggesting that the village's origins are Saxon.

It is not clear to whom the king granted Albury, or who held it until early in the 13th century when it belonged to William de Redvers, 5th Earl of Devon.

It remained in his family until 1293, when his granddaughter Isabella de Fortibus, Countess of Devon died.

It had a small nave and chancel, two Norman doorways and a 14th-century Decorated Gothic east window.

In 1828 the church was demolished and in 1830 a new St. Helen's was completed, designed by the Gothic Revival architect Thomas Rickman.