A fortified manor house was recorded at Biddlestone in 1415 and a survey in 1541 disclosed a pele tower with a barmkin in good repair in the ownership of Percival Selby.
In 1715, the house was described as in the ownership of Thomas Selby and comprised a cruciform four winged structure with a central battlemented tower.
A pele tower at Branxton, Northumberland then in the county of Islandshire, owned by William Selby was destroyed by the Scots in 1496 and was rebuilt by his son John (d 1565).
It did not remain the main family home as William had purchased Twizell Castle from Heron in 1520 and that estate was developed in preference to Branxton.
In 1644, the estate came into the ownership of a nephew, George Selby of London, who was appointed High Sheriff of Kent in 1648.
In the 18th century, the estate passed via the female line when Dorothy Selby married John Browne.
On the death of the 9th Viscount Montague in 1797 the Browne successors and descendant Thomas Selby of Ightham made an unsuccessful claim to the Viscountcy.
Oliver Selby, son of John, purchased half of the Manor of Beal, Northumberland in 1588 and land at neighbouring Lowlin in 1629.
His son Prideaux John Selby (1789–1867), was an eminent naturalist who improved the Twizell property and sold the Beal estate in 1850.
A great grandson, also named Prideaux Selby (1747–1813) became a colonial administrator in Canada, Another great grandson Henry Collingwood Selby (d 1839) bought an estate at Swansfield, near Alnwick where he built Swansfield House to a design by architect John Dobson in 1823.
Selby also commissioned the Camphill Column, possibly as a reaction to locals thought to be supportive of the French Revolution.
A later Prideaux Selby of Swansfield, a barrister, High Sheriff and Deputy Lieutenant of Northumberland, was also of Pawston.
By 1860, ownership of the estate had passed to the Beal/Holy Island branch of the family whose Prideaux Selby (1810–1872) had married in 1840 Sir Thomas Beauchamp-Proctor.