The castle was built by reforming churchman John Carswell, who was Rector of Kilmartin, Chancellor of the Chapel Royal at Stirling, and later titular Bishop of the Isles.

[1] Construction began in 1565 using masons brought from Stirling; although notionally built for Carswell's patron, the Earl of Argyll, he intended it as a personal residence for himself.

In 1690, he petitioned Parliament claiming Maclean clansmen burnt Carnasserie Castle, stole 2,000 cattle and murdered his uncle Alexander Campbell of Strondour.

Over the entrance are blank panels framed by carved supports, as well as the arms of the 5th Earl of Argyll with the Gaelic motto DIA LE UA NDUIBHNE, "God be with O'Duine", referring to the semi-legendary ancestors of Clan Campbell.

An archway bears the inscription SDC LHL 1681, for Sir Duncan Campbell, 4th Baronet and Lady Henrietta Lindsay, whose support for Argyll's uprising led to the castle's destruction.