James belonged to the noble House of Baux, specifically the branch settled in the Kingdom of Naples.
He was the son of Francis, Duke of Andria, and Margaret, daughter of Prince Philip I of Taranto and his second wife, the Empress Catherine II, Princess of Achaea.
[2] On the death of Philip II of Taranto in 1374, most of the barons in the principality of Achaea recognized as his heir Queen Joanna I of Naples.
James met with some success in 1380 but did not have complete control until Joanna's death in 1382, when he became the only legitimate claimant to Achaea.
[3] Early in 1382, James married Agnes, the widow of Cansignorio della Scala, lord of Verona.