Towards the end of the 18th century the owner of Forton Mill sold an adjacent parcel of land to the Board of Ordnance, to serve as the site for an Army General Hospital.

[2] The Barracks opened in 1807, consisting of four tall pavilions connected by arcades (an arrangement very similar to that of the contemporary military hospitals at Plymouth and Deal).

[1] The pavilions faced the main entrance gate (which was flanked by officers' quarters) across a sizeable parade ground, believed for a time to have been the largest in the country.

The officers' mess included space upstairs for 80 diners plus a 30-piece orchestra, with a library and breakfast room provided downstairs along with kitchens to the rear.

By 1862 additional barrack blocks had been built between the old pavilions so as to accommodate the full complement of over 1000 men;[3] further expansion, with the construction of married quarters for officers, took place in the 1890s.