Agler was ennobled to the rank of Ritter by the Habsburg emperor Ferdinand I and in 1539 received the title of an Edler of Paumgarten and Neuwaldegg.

In 1765 Field Marshal Count Franz Moritz von Lacy, confidant of Empress Maria Theresa and her son Joseph II, purchased the estate.

It included 17 reed cottages on the Hameau (French for "hamlet") hill to accommodate Lacy's guests, as well as a mausoleum built in 1794 that became his last resting place.

When he opened his gardens to the public, they became a popular destination for Vienna day-trippers.After Lacy's death in 1801, the princely Schwarzenberg family bought the palace and the large landscape park, which up to today is called Schwarzenbergpark, featuring two obelisks located on the 2.2 km (1.4 mi) long Schwarzenbergallee avenue, along with a number of statues of Greek gods.

However, the new owners had many art treasures transferred to Český Krumlov Castle in Bohemia, and over time the building and the gardens fell into a state of general neglect.