In 1769, Henry carried out substantial alterations to the then existing house to create the present two-story, four-bayed castellated Gothic Revival-style mansion.
Swinburne died in 1803 and in 1806 the property was sold to Anthony Surtees.
His son Robert Smith Surtees, a novelist, acquired the estate in 1838.
He died in 1864, leaving his estate to his daughter Eleanor, who married John Gage Prendergast Vereker, 5th Viscount Gort in 1885.
Their first son John Vereker, 6th Viscount Gort VC was succeeded in 1946 by his brother Standish Vereker, 7th Viscount MC who lived at Hamsterley until his death in 1975.