He had seven brothers and sisters: Amy Sophie (1863–1947), Florence Louise (1865–1891), Edith Mary (1866–1877), Herbert "Bertie" Thomas (1868–1928), Maud "Cissie" Helen (1870–1940), George Arthur (1874–1919) and William Lacy (1877–1902).

After his father's death, Sullivan's uncle Arthur visited the family often and became guardian to all of the children.

[2] In December 1883, at the urging of Charlotte's brother, William Lacy, the family emigrated to Los Angeles, California, in the United States, except for Sullivan's eldest brother, Bertie, who stayed with their uncle Arthur in London.

[3] Charlotte died in January 1885, barely a year after the move to the U.S. Sullivan's stepfather, Hutchinson, unable to cope with this loss and his responsibilities, soon returned to England, leaving the six surviving children (Edith had died) to be brought up mostly by their uncle William, with the financial support of their uncle Arthur.

[6] Sullivan married Kate Webb in 1902, and the couple had two daughters, Helen Mary (1906–1966) and Sheila (born 1919).