Ca' Vendramin Calergi was designed in the late 15th century by Mauro Codussi, architect of Chiesa di San Zaccaria and other noteworthy churches and private residences in Venice.

In 1581, the Loredan heirs of Andrea sold it for 50,000 ducats to Eric II, Duke of Brunswick-Lüneburg who took loans to afford it and to host sumptuous dinners for the Venetian nobility.

However, the duke kept it for only two years before selling it to Guglielmo I Gonzaga, Marquis of Mantua, who then sold it to Vittore Calergi, a Venetian noble from Heraklion on the island of Crete.

Calergi greatly expanded the building in 1614 with a large addition by architect Vincenzo Scamozzi called the "White Wing" which included windows overlooking a garden courtyard.

In the turmoil of the Risorgimento, they were forced to sell the palace to Caroline's grandson, Henry (Enrico), Count de' Bardi, and many of its fine works of art were auctioned in Paris.

Count de' Bardi and his wife Infanta Adelgundes and the related Dukes of Grazia maintained the home and hosted many famous names of the day.

He rented the entire piano nobile (mezzanine) level of the Ca' Vendramin Calergi from Count de' Bardi before his departure and arrived on 16 September 1882 with his wife Cosima Liszt, four children (Daniela von Bülow, Isolde, Eva and Siegfried Wagner) and household servants.