His genealogy can be traced back to the earliest settlers in the colonies, including Edmund Rice, a founder of Sudbury, Massachusetts.

[3][4] Another direct ancestor, William Sprague, left England on the ship Lyon's Whelp for Plymouth/Salem, Massachusetts.

His father contracted typhoid fever while DeDe Ball was pregnant with Fred, and died in February 1915.

Their grandfather, Fred Hunt, was an eccentric socialist who enjoyed the theater and frequently took the family to vaudeville shows.

In 1927, a neighborhood child, Warner Erikson, was paralyzed by a shot accidentally fired from a gun Fred's grandfather had given him for his birthday.

He left Jamestown, along with his mother and grandfather, to join his sister in California in the 1930s, working as a page boy at Cafe Trocadero.