The facility had its origins in a dispensary for the sick poor which was established in 1801 in the town centre near the harbour, initially in Manse Lane, then nearby in Cathcart Street.

In 1806 a contagious fever spread from the crew of a Russian prize ship in the harbour, causing many deaths, and one of the surgeons proposed a new hospital or fever-house, "where the poor would be removed from their own uncomfortable dwellings, not only for their own sakes, but for the purpose of checking infection.

"[2][3] In 1807, funds were raised to build a Hospital or Infirmary, and a site obtained on the south edge of the town, up on the east side of Inverkip Street between the Anti-Burgher Secession Church of 1803 (called the Canister Kirk for its shape) and the 1789 cemetery.

[3][2] The building on Inverkip Street was designed by the local harbour engineer, John Aird, the foundation stone was laid in 1808, and the first patient was admitted in June 1809.

[5] The infirmary was extended with wings added in 1830 after another fever epidemic,[4][6] and an 1847 extension incorporating the site and parts of the Canister Kirk continued the hospital round into East Shaw Street.