His mother's family, the Afan de Ribera were located in Sevilla and Cádiz and were involved in the slave trade and the settlement of the Canary Islands.
As Viceroy, he blocked the promulgation of the decrees of the Ecumenical Council of Trent and also Philip II of Spain's wish to introduce the Spanish Inquisition into the Kingdom of Naples.
He married Leonor Ponce de Leon, but did not father any legitimate children with her but fathered at two known illegitimate children: His appointment as Viceroy of Naples brought Afán de Ribera to Italy, where the collecting of Greek and Roman art was an important element of Renaissance culture.
He quickly began to imitate the aristocratic Italian collectors of the day, purchasing hundreds of works of ancient sculpture and other antiquities, which he shipped back to Seville to adorn his palace, the Casa de Pilatos.
In the 1960s the latter palace was sold and demolished and the collection was broken up, with some pieces going to the Marquesa of Cardona in Cordoba, and others eventually returning to the Casa de Pilatos in Seville.