The original motte and bailey castle, the remains of which can be seen to the south of the churchyard, was built here in the Forest of Galtres by Bertram de Bulmer, Sheriff of York during the reign of King Stephen (c. 1135–1154).

[1] The stone castle was built at the western end of the village by John, Lord Neville in the late fourteenth century.

In 1377, John Nevill obtained a charter for a market on Monday and an annual fair on the eve of the exaltation of the Holy Cross (14 September).

[2] The building has been credited to John Llewyn, who also built nearby Bolton Castle in 1378, on stylistic and documentary grounds.

Ralph's grandson, Ralph Neville, 2nd Earl of Westmorland, retained the title and the Durham estates and his younger son by his second marriage, Richard Neville, 5th Earl of Salisbury, retained the Yorkshire estates, including Sheriff Hutton.

In July 1484, Richard established the Council of the North, with its chief headquarters at Sheriff Hutton and Sandal Castle.

[9] In 1485, while awaiting the invasion of Henry VII at Nottingham, Richard sent his niece, Elizabeth of York, her sisters, and the Earls of Warwick, Lincoln, Lord Morley and John of Gloucester, to the castle.

[21][22] The castle is quadrangular in form, with four rectangular corner towers connected by ranges of buildings, enclosing an inner courtyard.

The northern and western sides are straight, whereas those on the south and east contain obtuse, outward pointing angles at their centres.