In the 19th century, the collection had become so large (and the Agnietenkapel so derelict) that the books were housed at several other locations, until 1881 when the library moved to the Handboogdoelen at Singel 421, the former home of the Long-bow militia.

During restoration works in 1968, much of the original 16th-century building reappeared and it turned out that the Library probably contains some of the oldest masonry in town and the only fireplace with a late medieval mantelpiece.

Queen Wilhelmina, who had fled to London at the outbreak of the war, gave her written permission to add the building to the Library.

In 1939 the church was demolished and the plot left vacant until the second half of the Sixties, when the new building by architect Jan Leupen was built.

The Special Collections at Oude Turfmarkt are housed in two adjacent buildings: one from 1642, designed by Philips Vingboons, the other from 1842-1843 is the former St Bernardus mental home.