The land on which the church stood now forms a significant part of Victoria Square.

[6] The foundation stone was laid on 22 July 1805 by the Lord Chamberlain, George Legge, 3rd Earl of Dartmouth.

The building was designed in stone in the Classical style with Doric columns dominating the west front.

The original design had a cupola instead of a spire,[7] but funding delays resulted in plans being redrawn.

[8] There were also a large number of tiered stone vaults several stories deep, divided into catacombs, underneath the church.