The castle was in the possession of William de Avenel in the mid-12th century, before passing to the Clan Graham by marriage and was then passed to the Clan Douglas by marriage.

[3][4] On 18 April 1455, the castle was captured and destroyed by King James II of Scotland, after a siege against James Douglas, 9th Earl of Douglas who had lost the support of James Hamilton, 1st Lord Hamilton.

Many of the senior members of garrison were hung and the lands passed to Clan Seton.

[3][5] Parts of the castle are thought to have been re-used in the 15th or 16th century.

[6] The site is now found within the grounds of Hopetoun House.