It stayed in his family until approximately 1320 when his descendant, William de Soulis, forfeited it because of suspected witchcraft and the attempted regicide of King Robert I of Scotland.

Legend has it that Soulis's tenantry, having suffered unbearable depredations, arrested him, and at the nearby Ninestane Rig (a megalithic circle), had him encased in lead sheet and boiled to death.

King James IV was suspicious of the then Earl of Angus, Archibald, Bell the Cat and his relationship with Henry VII of England, and ordered him to relinquish The Hermitage to the Crown.

Mary, Queen of Scots, made a famous marathon journey on horseback from Jedburgh to visit the wounded Bothwell there, only a few weeks after the birth of her son.

After Mary's forced abdication following the confrontation at Carberry Hill, Bothwell, facing charges of treason, fled to Norway and his titles and estates were forfeited by Act of Parliament.

Whilst attempting to raise an army to restore Mary to the throne, he was arrested by King Frederik's men for breach of marriage contract with Anna Throndsen, and imprisoned at Dragsholm Castle in Denmark, where he died insane and in appalling conditions.

The Scotts are descended matrilinearly from the Douglases of Drumlanrig, a cadet branch, and sometimes use the surname Montagu-Douglas-Scott, thus maintaining a continuity with earlier times.

It is now cared for by Historic Environment Scotland, an executive agency of the Scottish Government, and is open to visitors from 1 April to 31 October each year.

[5] Hermitage Castle is the subject of local ghostlore stories where it is reputed to be haunted by both Redcap Sly, de Soulis's familiar spirit,[6][7] and by Mary, Queen of Scots.