Buried in one of the lawns is a stone coffin containing what were thought to be the remains of Thomas, 2nd Earl of Lancaster who was beheaded in 1322 on the orders of King Edward II.
The coffin was excavated in March 1882 from a nearby field and re-interred at Fryston Hall at the request of the then owner.
The Crowle family developed coal mining on the estate but in 1788 the estate was sold to Richard Slater Milnes (1759–1804), the heir to a cloth fortune and MP for York, who improved the house and planted many trees.
From him it descended to Richard Monckton Milnes, 1st Baron Houghton (1809-1885), the poet, writer and unsuccessful suitor of Florence Nightingale, and also an MP for Pontefract.
The partially derelict hall was then demolished in 1934 and some of the stone used to build the Holy Cross Church at Airedale.