Lord Holland was the eldest son of Henry Fox, 1st Baron Holland of Foxley (1705–1774) of Holland House and his wife Lady Caroline Lennox (1723–1774), suo jure 1st Baroness Holland of Holland, a daughter of Charles Lennox, 2nd Duke of Richmond.
Stephen and his younger brother, the great Whig statesman Charles James Fox (1749–1806), were a great trial to their parents because of their gambling and other habits.
When his father died on 1 July 1774, Holland inherited his title (Baron Holland of Foxley) and then his mother's title (Baron Holland of Holland) upon her death three weeks later.
Holland died just over four months later of dropsy at Red Rice, Hampshire.
On 20 April 1766 he married Lady Mary FitzPatrick, a daughter of John FitzPatrick, 1st Earl of Upper Ossory with whom he had two children: