Only the ruins of a single tower and a section of curtain wall survives,[1] of which is associated with Greys Court, a Tudor country house.

The earliest mention of a manor at Rotherfield Greys, then known as Retherfield, was in 1283 during the reign of King Edward I.

The current castle at Rotherfield Greys, which was probably not built for defensive purposes, replaced this manor and it was built circa 1347 after being crenelated by Johannes de Grey de Retherfeld on 10 December 1346.

[2] The east front, which was destroyed during the English Civil War, was extended around 1600 by William Knollys and the castle was slighted during the English Civil War; it has been in ruins since c. 1646.

[citation needed] It is a Grade I listed building, along with the rest of Greys Court.