Construction of the earliest parts of the castle, the Upper Ward including the Queen Elizabeth Gate, began in 1594.

The destruction of the medieval Abbey church in the heart of the castle complex which had been used as the storehouse for ammunition and provisions forced Carteret to surrender on 15 December 1651 after being besieged for seven weeks.

French troops under Baron Phillipe de Rullecourt landed in St Helier on 6 January 1781, and the castle garrison was marooned.

A plan to link the castle to the mainland as part of an ambitious harbour project in the 19th century was abandoned.

During the Second World War the Germans, who occupied the Channel Islands, modernised the castle with guns, bunkers and battlements.

Each year, on the Sunday closest to St. Helier's Day, 16 July, a municipal and ecumenical pilgrimage is held to visit the Hermitage.

Among the historical displays is the regimental museum of the Royal Jersey Militia that holds several centuries of military memorabilia.

There is also a museum that discusses the evolution of cannons and fortifications that holds several pieces from the nineteenth century, and earlier.

Every Sunday through the season when the castle is open, a team of Historical Interpreters recreate the garrison of 1781, at the time of the battle of Jersey.