The collection was created by Duke Albert with the Genoese count Giacomo Durazzo, the Austrian ambassador in Venice.

Count Durazzo, who was the brother of Marcello Durazzo, the Doge of Genoa – "wanted to create a collection for posterity that served higher purposes than all others: education and the power of morality should distinguish his collection...." In the 1820s Archduke Charles, Duke Albert and Maria Christina's foster son, initiated further modifications to the building by Joseph Kornhäusel, which affected mostly its interior decoration.

In early 1919, the new socialist government of Austria confiscated, without compensation, both the building and the collection belonging to the Archduke Friedrich and evicted him.

Modifications of the exterior entrance sequence, including a distinctive roof by Hans Hollein were completed in 2008, when the graphics collection finally reopened.

In 2018, the Albertina acquired the Essl Collection of 1,323 contemporary artworks, including pieces by Alex Katz, Cindy Sherman, Georg Baselitz, Hermann Nitsch, and Maria Lassnig.