The building, which dates from 1766 in its original incarnation as the ballroom of Brighton's most fashionable Georgian-era inn, has been used for many purposes since then, and now stands 1 mile (1.6 km) away from where it was built.

Brighton began to develop as a spa town and seaside resort in the mid-18th century, encouraged by local doctor Richard Russell's influential advocacy of the therapeutic use of seawater, by drinking it and bathing in it.

[1][2][3] These activities became fashionable among high society and rich people, which gave the declining fishing village a surge of popularity in the 1750s.

[11] In 1815, Shergold offered a 25% share in the building and its land, and the Prince Regent (later King George IV) bought it through an intermediary, Thomas Attree, for £1,960 (£183,400 in 2025).

[10][14][15] The ballroom was converted into the recently completed Royal Pavilion's private chapel for the Prince Regent—who by this time was King—and was consecrated on 1 January 1822 by the Bishop of Chichester.

[10] Designer William Tuppen was responsible for the interior refit, which included the conversion of the musicians' gallery into the King's own pew and the installation of an organ supported by Gothic-style columns.

[20][22] The church was opened for public worship under its new name, St Stephen's, on 25 July 1851, and was consecrated on 11 June 1852 by the Bishop of Chichester, Ashurst Turner Gilbert.

[27] Crunden's interior, which had survived largely intact since he built the ballroom in 1766,[20] was damaged by fire soon after the day centre opened, but it has been restored.

[27][29] John Crunden designed the Castle Inn ballroom in the Adam style, which is still discernible in the interior despite the many changes of use the building has experienced.

[27][11] The internal walls had elaborate pilasters decorated with scrolls and friezes, and at the north and south ends there were recessed areas separated from the main section by columns.

[23] In contrast to the Adamesque interior, the church was given a plain stuccoed Classical frontage[30] facing Montpelier Place, with Doric pilasters below a pediment and cornice, topped by a lantern.