Also in that year, with others, he abducted Peter of Aigueblanche, the Bishop of Hereford, confining him to Eardisley Castle.
After Lewes, Giffard joined the retinue of Gilbert de Clare, earl of Gloucester and Hertford, and his association with the powerful Marcher Lord was rewarding in money and property.
He caused a scandal, for which he was heavily fined, in 1271 when he abducted her while negotiations for the marriage were in progress.
Giffard married secondly, in 1286, Margaret, widow of John de Neville, of Hallingbury, Essex.
Their son John (1287–1322) was executed by Edward II as a rebel, and Brimpsfield Castle was destroyed.