During the French occupation, the old men still living in the hofje were moved a block away to the present-day Proveniershuis, when the art collections of the two institutions were merged.

The older pieces of the museum collection, consisting of primarily religious themes, are Haarlem relics from the Reformation, when all Roman Catholic art was formally seized by the city council in 1648.

The first signs of an official museum with a curator occurred when the Dutch Society of Science, founded in 1752, started to rent the Prinsenhof room of the city hall in 1754 for its meetings and began to furnish it as a Cabinet of curiosities.

From an inventory list in the city archives it can be seen that they used as a model for their system of naming and presentation, the book Amboinsche Rariteitkamer by Georg Eberhard Rumphius.

[3] The spacious room soon proved too small for the number of donated artifacts it received from its members, thanks to the increase in shipping and associated travel.

In 1777 the Society moved its overflowing collection to a renovated house on the Grote Houtstraat, where the new young curator Martin van Marum would live the rest of his life.

Without an official curator, the painting collection was only available to be seen by appointment with the city clerk, a situation that has remained up to the present day for the large pieces still located there, such as the whalebone from Willem Barentsz trip to Nova Zembla or the portrait of Kenau Simonsdochter Hasselaer.

The art critic Victor de Stuers was very angry about Haarlem being the location of such museums, as there was no artistic climate there to speak of.

[4] Thus this antiquated collection is the one that was transferred to the Groot Heiligland in 1913, and large pieces that were not in the cloisters at that time, such as the painting by Dirck Ferreris installed in the mayor's room, remained at city hall.

The museum celebrated its 100th anniversary on the Groot Heiligland in 2013 with a Frans Hals exhibition that included reproductions being placed around the city in original locations.

In the late 19th century the museum became something of a pilgrimage site for young impressionists, who were fascinated by the loose brushwork visible in the two group portraits of regents by Hals that he painted when he was in his eighties.

This is the reason that after the move to the present location in 1913, the museum took on the name of Frans Hals as these were considered the most prominent paintings of the collection at the time.

Most of the objects and paintings can not be displayed for lack of space, and the museum rotates its collection through exhibitions at various locations in Haarlem, though works by some prominent painters cannot be lent out and remain in storage.

Famous modern artists such as Manet and Van Gogh even travelled to Haarlem to admire his group portraits of bailiffs and regents.

The windows have been decorated with art by anonymous Haarlem glass artists, most of which has been acquired through municipal demolitions work.

Spread along the corridors are beautiful Dutch tiles from local salvage operations that have been installed along the walls, accompanied by 17th century furniture including clocks, chairs, and chests.

More art has survived up to today from that period in Haarlem than from any other Dutch city, thanks mostly to the Schilder-boeck published by Karel van Mander there in 1604.