The present castle was built about 1160 by the Umfraville family at the request of King Henry II on land awarded to them following the Norman Conquest, presumably as a defence against the Scots.

In 1296, it was besieged by Robert de Ros, and some 40,000[citation needed] men, but the siege by the supporters of John Balliol was withstood.

In 1515, Margaret Tudor, the widowed queen of King James IV of Scotland and sister of King Henry VIII of England, having been banished by the regent, John Stewart, Duke of Albany, came to the castle with her second husband, Archibald Douglas, 6th Earl of Angus.

Margaret Douglas was to become the mother of Lord Darnley, the second husband of Mary, Queen of Scots, and grandmother of King James VI of Scotland and I of England.

[4] The castle became a royal property in 1546, and repairs were supervised by Sir John Forster, warden of the Middle Marches.