The son of James Butler, 1st Earl of Ormond and Lady Eleanor de Bohun, James was born at Kilkenny Castle and given in ward on 1 September 1344 to Maurice FitzGerald, 1st Earl of Desmond for the fine of 2,306 marks; and afterwards to Sir John Darcy who married him to his daughter Elizabeth.
He was usually called The Noble Earl, being a great-grandson, through his mother, of King Edward I of England.
In 1364 the Irish House of Commons sent a delegation to England, headed by Kildare, to complain of misgovernment, and to ask for the removal of "corrupt" officials, some of whom had links to Ormond.
He was Lord Justice by 24 July 1376, with a salary of £500 a year, in which office he was continued by King Richard II of England.
He died on 18 October 1382 in his castle of Knocktopher (near which he had, in 1356, founded a Friary for Carmelite friars).