The gallery is located in the building of a former 18th century Jesuit monastery in the historic area of Gradec in Zagreb's Upper Town, in the vicinity of other landmarks such as St. Catherine's Church, St. Mark's Square, Old City Hall, and Lotrščak Tower.

[1] Formally established in 1980 as a venue to house Ante Topić Mimara's extensive art collection, the former Jesuit monastery was adapted into an exhibition space in the following two years.

But since the controversial collector was not satisfied with the choice of venue for his artworks,[2] the gallery first opened to the public in March 1982 with shows that included major traveling exhibitions of international importance, such as collections of engravings by Albrecht Dürer, and works by painter Oton Gliha and sculptor Dušan Džamonja.

[3] The gallery's biggest success in its early years was a 1984 exhibition of ancient Chinese art, which featured 160 objects loaned from the National Museum of China and attracted hundreds of thousands of visitors over three months.

[3] However, the remaining two venues in Zagreb's historic Upper Town continued to function, with GKD and Lotrščak Tower hosting various kinds of travelling exhibitions of both Croatian and international artists.