James IV returned on 24 August 1513 on his way to besiege Norham Castle, and held a council or parliament at "Twesil" or "Twesilhaugh.

[2] Of the medieval structure, blocked windows, a chamfered doorway and the original north-east angle quoins are all that remains visible now.

From about 1770, Sir Francis Blake (d. 1780) worked on the recreation of the castle as a Gothic Revival mansion, designed by architect James Nesbit of Kelso to be five levels tall.

When in 1882, the Blakes built a new mansion at Tillmouth Park much of the incomplete Twizell Castle was demolished and the stone used in the new construction or sold.

Rectangular in plan, with circular towers on the angles and two wings on the north side, the basement rooms in the main block are stone and brick-vaulted as a precaution against fire.