Roger de Pont L'Évêque, Archbishop of York from 1154 to 1181, rebuilt the minster, and appears to have also begun the construction of the palace.
This was supposedly to purchase Grays Court for his son, George, the building probably being extended in part with stone taken from the palace.
He rebuilt part of the palace as his own house, and laid out gardens in the remainder of the grounds, which include a fishpond, tennis court and bowling green.
Charles I of England stayed in the house in 1642, but after Ingram's death, his descendants divided the property into small tenements and let them out.
In 1780, the former great hall of the palace was rebuilt as a riding school, noted in 1785 as the site of an early balloon ascent.