The castle overlooks the harbour and although pre 14th century in construction, the tower dates back to 1494 and the visit of James IV to the Western Highlands.

It was reinforced with the addition of an outer bailey and towers in the 1320s by Robert the Bruce.

A tower house was added in the 16th century, which is the most noticeable part of the remains.

The castle occupies high land above the village, providing views up Loch Fyne and beyond to the Firth of Clyde.

In 1687 the castle was involved in another skirmish when Walter Campbell of Skipness Castle seized it as a stronghold for Archibald Campbell, 9th Earl of Argyll as part of actions in support of the Monmouth Rebellion in England.