Documentary evidence from the late 14th century records it as Dounarwyse Castle in the possession of the Lords of the Isles.

When they backed the losing side in the dispute between Edward I and Robert de Bruys, their lands were declared forfeit and the castle was given by King Robert to Angus Og of Islay Lord of the Isles and Clan Donald.

During the late 14th and 15th centuries, it was used as an occasional base by the Lords of the Isles, at which time it was known as Dounarwyse Castle.

When, some centuries later, the latter tried to conquer Scotland, they too had their lands declared forfeit, and this time Clan Maclean gained the Castle at MacDonald expense.

Lord Ochiltree entertained the Island Chiefs here in 1608 before making them prisoners.