He was the eldest surviving son of Alexander Stewart, 4th High Steward of Scotland (d. 1283), by his wife of unknown identity.

For these reasons, and also the fact of his son and successor Walter Stewart, 6th High Steward of Scotland being described in about 1314 as a "beardless lad" by John Barbour in his poem The Brus, it is proposed that James was born in about 1260.

He subsequently submitted to King Edward I of England on 9 July 1297 and was one of the auditors for the competitor, Robert Bruce, 5th Lord of Annandale.

After the victory of England against Scotland, Stewart would be compelled to swear fealty to King Edward again at Lanercost Priory on 23 October 1306.

To render his oath inviolable, it was taken upon the two crosses of Scotland most esteemed for their sanctity, on the consecrated host, the holy gospels and on various relics of saints and he agreed to submit to instant excommunication if he should break his allegiance to Edward.