The newly appointed Privy Councillor Magnus Julius De la Gardie, who was in possession of a great fortune through his marriage to Hedvig Catharina Lillie, commissioned architect Joseph Gabriel Destain to design the present palace, built between 1720 and 1727.

Between 1778 and 1793, Frederick Adolf resided there with his lover Sophie Hagman, and many episodes from this period are preserved as the Tullgarnsmminnena, The Tullgarn memories.

The following year, it was granted to the heir to the throne, the future Oscar I of Sweden, and served as the summer residence of the Swedish royal court during his reign.

Much of the present interior dates from the time of King Gustaf V and Queen Victoria, including the vestibule, whose walls are covered in hand-painted Dutch tiles.

The breakfast room is furnished like a south German Bierstube, possibly reflecting the fact that Queen Victoria came from Baden in Southern Germany.