His father had fought in the War of the Castilian Succession on the side of the future Queen Isabella I of Castille against her niece Juana la Beltraneja, and his mother was the younger half-sister of Juana Enríquez y Fernandez de Cordoba, making him the first cousin of Ferdinand II of Aragon.

He participated in the conquest of Granada, and already as Duke, he led the Spanish army against the French in Roussillon in 1503.

When Ferdinand II as regent of Castilla, decided to invade and conquer the Kingdom of Navarre, supported by a Papal bull, he put the Duke of Alba at the head of his army.

He accompanied the Emperor to Germany, Flanders and Italy and was made in 1520 a grandee of Spain and a knight in the Order of the Golden Fleece.

His eldest son García was destined to be his successor, but predeceased his father, passing the title of third Duke of Alba to Fadrique's grandson, the famous Iron Duke Fernando Álvarez de Toledo y Pimentel.