Born in 1421, possibly at Lizy in Picardy, he was brought up in the Burgundian court with his younger half-brother, the Count of Charolais, later Charles the Bold, last of the Valois Dukes of Burgundy, to whom he grew very close.

In 1466, he was present with Charles at the siege of Dinant, and in the same year he was invited by King Edward IV for a stay in England during which he jousted against Anthony Woodville, 2nd Earl Rivers, the queen's brother, in a famous contest which spread over two days.

In contrast to his rather ascetic younger half-brother Charles, it seems that Anthony inherited his father's sexual proclivities: at the chapter of the Golden Fleece held in 1468, he was castigated for his fornication and adultery, in spite of his "valour, prowess and prudence and several other good habits and virtues".

In the middle of these travels, he managed to find time to call in at the siege of Neuss, and later that year he participated in the conquest of the Duchy of Lorraine.

In 1476–1477, he fought alongside Charles the Bold at the three great battles of Grandson, Murten and Nancy, and was taken hostage at the end of the latter by René II, Duke of Lorraine, and delivered to the King of France, who was anxious that Burgundy should never again rebel.