Het Provinciaal Genootschap van Kunsten en Wetenschappen in Noord-Brabant (Provincial society for arts and sciences) was founded in 1836.

The province supported the society with 1,000 guilders, and the municipality of 's-Hertogenbosch provided some rooms for the library.

The first location of the library of the Provincial society for Arts and Sciences was on the first floor of the Latin School on the Papenhorst.

[5] In 1868 the society was forced to move to new, but less suitable accommodations on the upper floor of the Butter hall on the market.

Pape gave the society the means to realize its ambition to create a real museum.

It started to look for a building, and found the Big Arsenal (Groot Tuighuis), which had been left by the Dutch military after World War I.

For the arsenal function, multiple solid floors had been created inside, making it a perfect place for a museum with a large library.

The authorities agreed to hand it over for a small lease of 200 guilders, provided that Oscar Leeuw, architect of Museum Kam in Nijmegen would change it to a suitable building.

Behind the hall the choir housed the stately meeting room of the society, with the painting 'Samson and Delilah' by Abraham van Diepenbeeck.

For those that were related to literature, archaeology, (local) history, or works by established painters, the society could count on its own volunteers.

[10] That same year the exhibition 'Noord-Brabant' showed books, etchings and paintings related to, or made by people from Brabant.

Meanwhile, a group called 'Friends of the Society' (Vrienden van het Genootschap) started to buy modern art for the museum.

It was also clear that the position of the 'elitist' Society for Arts and Sciences, which had paid many bills, would suffer if other means became available.

[19] In September 1967 a large exposition of Hieronymus Bosch (c. 1450 – 1516) was held in the museum on account of the 450th anniversary of his passing.

[20] It kindled a desire to have a city museum dedicated to art exhibitions, just like the Van Abbemuseum.

As regards the most famous North Brabant artists (Bosch, Van Gogh and the Brueghels), the museum was established too late to have a good collection of their work.

In January 2020, the museum bought 'head of a woman' (Dutch: Kop van een Vrouw) for 1.6 million Euro.

[24] By 2020, the museum's Van Gogh collection gives a good overview of the works that Vincent made in North Brabant.

[2] Notable exhibitions held at the museum include, Hieronymus Bosch - Visions of Genius.