In 1854 the community moved to its present Brompton Road site, adjacent to the Victoria and Albert Museum.

The site had previously been owned by Robert Pollard, who had set up a boys’ boarding school there, known as Blemell House.

[1] An attempt early in 1853 by the Vicar of Holy Trinity, Brompton, to prevent the establishment of a Catholic community so close to his church was unsuccessful.

The house contained the congregation's chapel, known as "the Little Oratory", decorated in 1871 by John Hungerford Pollen, who, at that time, was connected with the nearby South Kensington Museum (now the V&A).

Together with their Church of the Immaculate Heart of Mary, the community of the Oratorian Fathers is often popularly, though less accurately, referred to as the 'Brompton Oratory'.