It was the caput of the feudal barony of Clifford, a Marcher Lordship (owing allegiance directly to the king, but separate from the rest of the kingdom).

The early motte-and-bailey castle was built on a cliff overlooking a ford on the River Wye in 1070 by William FitzOsbern, 1st Earl of Hereford, on a wasteland formerly occupied by Browning.

The position of the castle as adjacent to the River Wye enabled the seasonal flooding of the river to fill the flood plain around the castle, forming a shallow lake or marsh with the assistance of a dam on the western or upstream side of the site, acting as a further form of defence.

After FitzOsbern was slain in battle at Flanders, the castle passed to his son, Roger de Breteuil, 2nd Earl of Hereford.

As the bulk of Ralph's time was spent in Normandy, the castle was rented to Gilbert, Sheriff of Hereford for 60 shillings.

Walter Clifford then made his peace with the Crown and led his troops against Prince Llywelyn ab Iorwerth, his father-in-law.

Findings from the excavations of this period unrelated to the castle included a boar's tusk, wolf vertebra and a Roman brooch.

[1] Between 1950 and 1953 a further series of excavations were embarked on, discoveries from which included the foundations of a tower on the motte, alongside further sections of curtain wall.

This work has now been completed and images can be seen at cliffordcastle.org Clifford castle now consists of a great motte as constructed by the men of William FitzOsbern in the late 1060s.