The building was commissioned and paid for by the local lord of the manor, Nathaniel Cholmley, and was designed and built, in the neoclassical style, by the architect Jonathan Pickernell,[1] who also constructed the two inner piers in Whitby Harbour between 1781 and 1812.

[4] The clock tower also holds a bell which was tolled to start any court process.

[5] The Old Town Hall served as the location for the first meeting of the Whitby Literary and Philosophical Society in September 1823.

[6] The society was started to provide funding and space to display fossils found in the environs of Whitby town.

[13] The building was given listed status in 1954[14] and was renovated in 1987 but is prone to decay with some stone falling off the top of one of the columns.