Located at NS086646, the castle has been described as "one of the most remarkable in Scotland",[1] for its long history dating back to the beginning of the 13th century, and its unusual circular plan.

According to Hákonar saga Hákonarsonar, Óspakr's forces fought for three days to take the castle, breaking down part of the eastern wall by hewing the stone with their axes.

Although the Battle of Largs is called indecisive, Hákon's campaign was a failure, the Norse retreated, and it effectively ended Norwegian authority in western Scotland.

Robert II granted the hereditary keepership of the castle to his son John, ancestor of the Earls and Marquesses of Bute.

Robert III made his eldest son David Duke of Rothesay in 1401, beginning a tradition of honouring the heir to the throne of Scotland with this title.

Construction of a gatehouse keep, extending from the north of the curtain wall, began around the turn of the century, to provide more modern accommodation for James IV.

The curtain wall itself was raised up to ten metres in height, the works continuing into the reign of James V. In 1527 the castle withstood another siege by the Master of Ruthven, which destroyed much of the burgh of Rothesay.

Esmé Stewart, 1st Duke of Lennox, a favourite of James VI was compelled by the Ruthven Regime to leave Scotland.

Following a long period of neglect, the 2nd Marquess of Bute employed 70 men to excavate the ruins, clearing large amounts of rubbish from the castle in 1816–17.

His "restorations" continued until 1900, and include the clearing and shaping of the moat, as well as the red sandstone additions to the forework, which reinstated the hall roof while significantly altering the character of the building.