He was the son and heir of Nicholas Audley, 1st Baron Audley (1289–1316) by his wife Joan Martin (died Feb. 1320 / 1 Aug. 1322)[clarification needed], who was the daughter of William Martin (died 1324), feudal baron of Barnstaple (in Devon), and Marcher Lord of Kemes (in what later became Pembrokeshire).

She was posthumously the eventual sole heiress of her brother William FitzMartin (died 1326) to Barnstaple and Kemes.

They had four children: three sons, Thomas, Roland (or Rowland) and James, who all died childless,[4] and a daughter, Margaret Audley (died 1373), who married Fulk FitzWarin, 4th Baron FitzWarin (1341–1374) of Whittington Castle, Shropshire and Alveston, Gloucestershire.

[5] Fulk's mother was said to be Joan de Beaumont, a daughter of Henry de Beaumont, 4th Earl of Buchan,[6] and was thus a sister of the wife of Fulk's half-brother-in-law Nicholas Audley, 3rd Baron Audley.

[4] James Audley had settled the feudal barony of Barnstaple by means of an entail on his heirs male, with remainder to the crown.