[2][3] The exact date of construction is unclear although the possibility of creating a castle at Rye was mentioned in documents of 1226 and 1249.

[5][6] During King Henry VIII's Device Fort programme, an artillery battery, known as the Gun Garden, was constructed adjacent to the castle overlooking the harbour.

[2] Ypres Tower is one of two sites of Rye Castle Museum and is a grade I listed building.

The East Street Site, a former brewer's bottling factory, is the main exhibit area for the Rye Castle Museum.

Opened in 1999,[13] the local history exhibits include fire fighting equipment, trade changes caused by the sea's retreat, maritime history and shipbuilding, antique toys and games, photos, town seals, and archaeological artifacts.