Joseph was born in London 21 April 1777, the youngest of the three sons, and three daughters of William and Elizabeth Fry.
However, their mother is credited with "the financial acumen which had enabled money both to be acquired and prudently managed: it was a quality which perhaps neither of the sons inherited".
He had five sons and six daughters (one of whom died in infancy):[3][5] During the 1812 financial panic in the City of London, William Fry precipitated a crisis, by lending a large amount of the bank's money to his wife's family, undermining its solvency.
The Gurneys acted as receivers and saved the tea merchant business, placing it under their control with Joseph Fry on a salary of £600 per year.
[11] After the death of Joseph's sister, Elizabeth Fry (1779–1844), they moved to her home, Plashet Cottage, East Ham.