A place of worship has existed on the hilltop site for about 1,000 years, but the present building has Norman internal features and a 13th-century exterior.

A wide range of monuments and wall paintings survive inside, including one commemorating Richard Shelley—owner of nearby Patcham Place and one of the most important noblemen in the early history of Brighton.

The church, which is Grade II* listed, continues to serve as the Anglican place of worship for residents of Patcham, which 20th-century residential development has transformed from a vast rural parish into a large outer suburb of Brighton.

Patcham's first church served a large rural area north of the fishing village of Brighthelmston—the ancient predecessor of Brighton.

A wide-ranging series of alterations were carried out by Victorian church restorers to improve the building's structural condition and provide more space to cater for the growing population.

The building's plain exterior contrasts with its well-preserved and, in parts, ancient interior whose features include wall paintings and stone memorials.

[4] William de Warenne, 1st Earl of Surrey, who held most of the land in the local hundreds, owned the manor.

[6][10] In 1898, at the same time as the north-side extension, the outside walls of the west, east and south sides were coated with grey cement, probably to improve their structural condition; although this has been described as "unsightly"[11] and "kill[ing] the exterior stone dead",[5] one historian has argued that because many medieval churches were rendered in this way, rather than having uncovered flint walls, it gives the impression of what a typical church of that era may look like.

[9] Patcham's proximity to the ever-growing resort of Brighton—it is 2 miles (3.2 km) north of the Palace Pier[8] on the English Channel coast—encouraged suburban growth from the mid-19th century.

From a low point of 286 in 1801, the population of the village steadily rose and had nearly quadrupled by the time of the United Kingdom Census 1901.

[3] The church contains several memorials, of which the oldest is a wall tablet commemorating Richard Shelley, one of the earliest owners of Patcham Place, who died in 1594.

[17][18] Shelley lived in Patcham from 1546,[3] and was an important figure in Brighton's early history: in 1579, he and three other local noblemen were appointed by the Privy Council to form a commission to record and regulate the "ancient customs" of the villagers and to mediate between the fishermen and the farmers, who often had conflicting needs.

[19] The memorial, of which only parts remain intact, is flanked by pilasters, the Shelley coat of arms and a naked grave-digger on each side.

[23] On the north side of the churchyard, which is traditionally used for burials of criminals and those who committed suicide, a memorial survives to "a smuggler, unfortunately shot".

[24] The plan of All Saints Church comprises a west tower with heavy buttresses and a broach spire, a three-bay nave with a tall aisle on the north side (described by one historian as "quite out of proportion to the rest of the building, particularly in height"),[5] and a porch on the south wall, a chancel and a vestry.

[7] Although the north aisle of 1898 was flint-built with courses of red brick, the rest of the church's exterior was clad in cement at the same time.

[12] It has been dated to c. 1230,[7][10][11] and depicts the ascension to Heaven of the dead and the weighing of their souls by Jesus Christ, who is flanked by the Virgin Mary and John the Baptist.

[7] The largest window, which sits under a hoodmould, is in the east end of the chancel; it is a three-light lancet with prominent tracery in the curvilinear/reticulated style.

[3][10] The west wall has a wide range of old carved prayer and commandment boards, which are a common feature of Sussex churches.

[7][9] Ancient examples of graffiti, consisting of etched crosses and dates, also survives inside; such "curious carvings" are also common in Sussex.

[15] The ecclesiastical parish of All Saints covers the north of Brighton and its rural hinterland, incorporating Withdean, Westdene, Patcham, Hollingbury, part of Preston (as far south as Preston Park railway station), and the scattered residential buildings north of the A27 Brighton Bypass and south of Pyecombe.