In 1326, Hugh was ordered by fugitive king Edward II to seize the properties of Henry, Earl of Lancaster.

He also supported the king by a long defense of Caerphilly Castle against the forces of the queen, which he only surrendered in 1327 after being promised his life and bodily integrity.

Following Edward III reaching majority and overthrowing his mother's faction, Hugh was freed and the king granted him lands and rents worth 200 marks annually.

Hugh died 8 February 1349 and was buried at Tewkesbury Abbey, Gloucestershire, the Barony le Despenser created for him becoming extinct.

Hugh's widow, Elizabeth, would remarry to Sir Guy, Baron Briene, before dying in 1359, also being buried at Tewkesbury Abbey.