For centuries this location was used to offload seagoing ships, and led to the growth of Port Glasgow close to the castle on either side and to the south.
In the late 16th century the castle was inherited by Sir Patrick Maxwell, a powerful friend of king James VI of Scotland and who was notorious for murdering two members of a rival family and beating his wife who left him after having 16 children.
In 1597 Sir Patrick expanded the building, constructing a new north range replacing the earlier hall in the form of a three-storey Renaissance mansion.
The mansion has features of the Scottish baronial style including crow-stepped gables and north corners embellished with corbelled turrets.
An early tenant was a ropemaker called John Gray who also dealt in wild animals such as big cats and bears which he obtained from ships visiting the Clyde and often housed in the castle cellars.