The palace was built some time between 1682 and 1684 for Adam Kotowski, the royal cup-bearer at King Jan Sobieski's court, and his wife Małgorzata Durant.

This large, three-storied Baroque building in Palladian style was designed by Tylman van Gameren.

In 1688 it was purchased by Queen Maria Kazimiera and transferred to the Benedictines of the Blessed Sacrament.

Around 1745 a new palace was built on the New Town Market Square and from 1754 to 1777, it was occupied by the Jesuit Collegium Nobilium.

[2] Between 1771 and 1779 King Stanisław August Poniatowski established a new building situated on the rear escarpment.