Construction of the building began in 1682 outside the walls of the city, on property belonging to the Tribunal of the Holy Office, the institution responsible for the Spanish Inquisition.

It was originally constructed as the seat of the University of Navigators (Universidad de Mareantes), a school to educate orphaned children and train them as sailors.

Atop the façade facing Calle Palos de la Frontera, across from the Hotel Alfonso XIII, are sculptures of twelve illustrious Sevillians, sculpted in 1895 by Antonio Susillo.

The twelve figures are: Three of these were Sevillians "by adoption", born elsewhere, but who lived and died in Seville: Benito Arias Montano was born in Fregenal de la Sierra (province of Badajoz), Rodrigo Ponce de León in Cádiz, and Juan Martínez Montañés in Alcalá la Real (province of Jaén).

The dukedom were not well received in the Spanish Court because Duke claimed for his wife part of the inheritance of Ferdinand VII.