Later the priest's official residence became the imposing Bramber mansion and landmark now called "Burletts" and located on Clays Hill.

Bramber was the caput of a large feudal barony held from the 11th to 14th centuries by the Braose family which was noted for its impact on the medieval history of the southern Welsh Marches.

The present building was constructed in about 1470 by William of Waynflete, Bishop of Winchester and founder of Magdalen College, Oxford.

The house has a music room which has two 14th-century ornately carved stone chantry tombs serving as fireplaces, and is regularly used for concerts and recitals.

King Charles II is claimed to have stayed at St Mary's House during his escape to France after defeat at the Battle of Worcester.

Just outside Bramber, in the direction of Botolphs village, formerly stood a medieval hospital and nunnery, caring for sufferers of leprosy, and dedicated to St. Mary Magdalene.