Before the Norman conquest of England, Horsford was held by an Anglo-Saxon named Edric but after the conquest William the Conqueror granted the land to Robert Malet, the Lord of Eye.

[1] Robert was exiled after he sided against Henry I, but had by then granted the manor of Horsford to Walter de Caen.

[1] Walter built a motte and bailey castle on the site, in an area of heathland, possibly positioning his castle to make a political statement about his control of these pasturelands.

When Robert and his wife were returning from a pilgrimage to Rome, they were attacked by bandits, robbed and thrown into prison.

All that now remains is the low-lying motte and bailey in a field to the east of the village.