Built in 1558 as part of the Patcham Place estate, it was owned for many years by Anthony Stapley, one of the signatories of King Charles I's death warrant.

The parish of Patcham, covering 4,325 acres (1,750 ha) of chalk downland north of Brighton, has Saxon origins, and the remains of farmsteads and intermittent settlements can still be discerned.

[2] At the time of the Domesday survey in 1086, William de Warenne, 1st Earl of Surrey—who owned large areas of land in present-day Sussex—held the manor of Patcham.

[4] Richard Shelley is believed to have lived in Patcham from 1546;[1] he was an important figure in Brighton's early history, because in 1579 he and three other local noblemen were appointed by the Privy Council to form a commission to record and regulate the "ancient customs" of the villagers and to mediate between the fishermen and the farmers, who often had conflicting needs.

During this time, he was the Governor of Chichester (the county town of West Sussex) for three years, became one of the 59 signatories of King Charles I's death warrant at his trial in 1649, and subsequently became a member of the English Council of State.

[1][6] Patcham Place stayed in the Stapley family until 1700, after which it passed through several owners (including George Nevill, 1st Earl of Abergavenny).

[1] During this time it was also used as the venue for local law enforcement, where the parish constable could take people suspected of crimes to be charged in the presence of a Justice of the Peace.

[7] In 1764, Major John Payne (or Paine) bought the house, and immediately set about extending and rebuilding it, eliminating all traces of the 16th-century structure.

[4][8] The northwest and northeast walls are hung with the glazed black mathematical tiles[8] which were a signature feature of residential buildings in Brighton between about 1760 and 1820.