It is situated on a promontory of a raised beach jutting out at a height of 60 feet (18 m) above coastal flats, with a level top defended by steep slopes or cliffs to each side of the point.

While Edward invaded the East of Scotland, in July he ordered a contingent including Robert the Bruce and Aymer de Valence to press up through Clydesdale, and meet a large seaborne force led by Bruce's father-in-law the Earl of Ulster which attacked Rothesay Castle then besieged Inverkip Castle.

[5][6] Bruce is said to have taken part in the fight to seize Inverkip Castle,[3] and then had the task of getting the siege engines north for the English assault on Stirling Castle: on 16 April 1304 Edward wrote to thank Bruce, referring in particular to the problem of finding "a waggon fit to carry the frame" of "the great engine of Inverkip".

[8] On 24 July 1306, after Bruce's defeat at the Battle of Methven, Edward gave orders that Sir Thomas Randolph was to be held prisoner in Inverkip castle, supervised by Gordon, with no option of bail: by September of 1307 he had apparently been freed.

Their son Sir John Shaw Stewart, 4th Baronet, commissioned a design for Ardgowan House from the architect Hugh Cairncross.