[3] When the second Baron Aldeburgh died in 1391 without issue, the castle transferred to the Ryther and the Redmayne (Redman) families, into which his two daughters had married.
At Cutler's death in 1693, it passed to his only surviving daughter, Elizabeth, Countess of Radnor and on her death without heir to Cutler's nephew, unmarried Edmund Boulter MP of Boston and Wimpole Hall then to his nephew John Boulter of Gawthorpe and Westminster, who died unmarried in 1738.
Centuries after it had been abandoned, Harewood Castle remained a landmark, the subject of several paintings in the 1797 by J. M. W.
[6] Recent times saw decay and weathering take their toll, and the unstable castle was placed on the Buildings at Risk Register by English Heritage.
As of 2008, the restoration project, which involved architects, geologists, structural engineers, ecologists and staff of the Harewood Estate, Historic Property Restoration Ltd and English Heritage, was nearing completion.