The palace was built about 1680, probably to Tylman van Gameren's design, for the high official and royal secretary Stanisław Kleinpolt.

From the 1730s the proprietor was Franciszek Maksymilian Ossoliński, and later Michał Kazimierz ("Rybeńko") Radziwiłł, who reconstructed the palace in the rococo style, probably to a design by Johann Sigmund Deybel.

The magnificent sculptures of the Four Winds (Notus, Boreas, Zephyrus and Eurus) atop the fence posts date from that period.

The right wing was widened, and a new annex was erected adjacent to ulica Długa, with an early-classicist front elevation.

In 1927 it was purchased by the Polish Treasury, restored, and made the seat of the Ministry of Labor and Social Welfare.