He was the son of Major-General Henry Ponsonby by his wife Lady Frances, daughter of Chambré Brabazon, 5th Earl of Meath.
He married a second time on 23 October 1752, to Louisa, daughter of Henry Lyons of Belmont.
After Sarah was orphaned by her father's death she was sent to stay with distant relations, Lady Elizabeth "Betty" and Sir William Fownes at the Woodstock Estate near Inistioge, County Kilkenny, where she attended school in Kilkenny.
She was deeply unhappy there and received unwanted attentions from Sir William, her guardian.
[1][2] By his third wife, Mary, daughter of Sir William Barker, 3rd Baronet of Kilcooly Abbey, he was the father of Mary, who married Thomas Barton of Grove House, Fethard, and of Chambré Brabazon Ponsonby-Barker.