Over the centuries, members of three distinct family dynasties have held the title in succession – the House of Álvarez de Toledo [es], the House of Silva (extinct in 1802) and the House of Fitz-James Stuart, which descends from an illegitimate son of King James II of England.

Famous holders of this dukedom include Don Fernando Álvarez de Toledo, 3rd Duke of Alba, governor of the Spanish Netherlands (references to "Alba" (or "Alva" in Dutch), particularly in the context of Dutch history, are usually about him), and Doña María del Pilar de Silva, 13th Duchess of Alba, a muse of the painter Francisco Goya.

When they first met, Francisco Goya was much older than the Duchess of Alba.

Today, the ducal family of Alba retains a large and valuable collection of art and historic documents.

The largest part of this treasure is kept at the main residence of the family in Madrid, the Liria Palace.