The town had passed to the Sieniawski family by 1706, when the palace and its surroundings were destroyed by Swedish troops during the Great Northern War.

Soon afterwards Maria Zofia Czartoryska married August Aleksander Czartoryski and between 1731 and 1736 they built a new Rococo palace on the site, to designs by Jan Zygmunt Deybel.

In 1706, when Pulawy became the property of the Sieniawski family, the Swedish army destroyed the palace and its surroundings during the Great Northern War.

It crossed in front of the palace entrance with a narrower avenue planted with trees (today's Czartoryskich Street), connecting the Lublin roadway with a winding gorge, the so-called Deep Road.

A two-flight external staircase adjacent to the alcoves, and in the upper section to the main body, led from the courtyard to the representative first floor.

The peak of development and functioning of the Enlightenment cultural center in Puławy falls at the turn of the 18th and 19th centuries, thanks to the multilateral activity of Izabela and Adam Kazimierz Czartoryski.

A large group of outstanding painters (Jean-Pierre Norblin de La Gourdaine, Zygmunt Vogel, Kazimierz Wojniakowski, Józef Richter), writers (Franciszek Dionizy Kniaźnin, Jan Paweł Woronicz, Julian Ursyn Niemcewicz), architects (Chrystian Piotr Aigner, Joachim Hempel), and musicians (Vincent and Franciszek Lessel), gathered at the Puławy manor house.

The heyday of the palace began in 1785, when Prince Adam Kazimierz Czartoryski and his wife Izabella née Fleming moved permanently to Puławy.

The reconstruction of the palace and the transformation of the park began in 1796, when Christian Piotr Aigner was the main designer of buildings in Puławy.

He also connected the palace with the left annex, building a neo-Gothic one-story orangery with a four-column portico and lions on the park side.

Attempts were made to take advantage of the natural qualities of the surroundings, e.g. grottos in the slope of the Vistula escarpment and an old tree stand.

A rotunda-shaped chapel was erected on the hill near the Lublin roadway, to which an avenue planted with four rows of trees led from the palace.

Prince Adam Jerzy Czartoryski, the owner of Puławy since 1812, was sentenced in absentia to beheading with an axe by the Tsar for his participation in the November Uprising, and all his possessions in the Russian partition were confiscated.

Zofia Zamoyska née Czartoryska took some of the sculptures (including "Tancred and Clorinda", a sarcophagus, lions, an obelisk dedicated to Prince Józef Poniatowski) to Podzamcze near Maciejowice, from where they returned to Puławy in 1947.

In the façade of the right wing of the palace, there is a stone plaque in memory of Krystyna Krahelska, who in 1940–42 worked as a laboratory assistant in the agricultural microbiology department of the Institute in Puławy.

On the left wing of the palace, near the passage to the so-called small park, there is a stone plaque commemorating Józef Piłsudski's stay in Puławy on 12–15 August 1920.

In the days preceding the Battle of Warsaw, the Marshal, together with a group of officers, worked out the last plans of the offensive against the Red Army in the column room of the library.

A memento of that time is still preserved in the column room, a hexagonal, supported on one leg, grand table at which Piłsudski's staff worked.

However, today's plaque is only an exact copy of the original, which was not found after it was probably taken down in 1942 on the order of Governor-General Hans Frank during his visit to the Institute.

In the arcades of the portico there is, among others, the oldest of the plaques commemorating Tadeusz Kościuszko, a pupil of the School of Chivalry, which remained under the command of Adam Kazimierz Czartoryski.