Built in the 18th century on the site of a decaying Quaker meeting house, it served Baptists in the town for many years until a new chapel was constructed nearby.

The medieval Cinque Port of Rye, on a sandstone hill in the middle of flat marshland,[1] has (in common with the rest of Sussex) supported a great variety of Christian denominations over the centuries.

[4] The religious census of Sussex in 1851 recorded that the chapel had 280 sittings, 150 of which were free; and attendances at morning, afternoon and evening services were given as 80, 60 and 140 respectively.

[3] Rye Particular Baptist Chapel, under its new name of Quaker's House, was designated a Grade II Listed building on 12 October 1951.

The roof, which has two dormer windows in the attic space above the first floor, is laid with tiles at the rear and slates at the front.