Before he was three, "Sunny Jim" experienced a transformational visit from his muse, Hermy, which he describes in his autobiography, Coming Unbuttoned (1993):[3] I remember waking in the dark and hearing my parents arguing in the next room.

If I followed the game sharply enough, I could be a useful spokesman for Big Joy.Broughton was kicked out of military school for having an affair with a classmate, and attended Stanford University before dropping out just before his class graduated in 1935.

In 1965, Broughton collaborated with harpist Joel Andrews to produce The Bard & the Harper, an album of recited poetry and music, on Gleeman Records.

With Singer, Broughton traveled and made more films – Hermes Bird (1979), a slow-motion look at an erection shot with the camera developed to photograph atomic bomb explosions, The Gardener of Eden (1981), filmed when they lived in Sri Lanka, Devotions (1983), a study of male relationships, and Scattered Remains (1988), a tribute to Broughton's poetry and filmmaking.

[11] Broughton put off marriage until the age of 49, when he married Suzanna Hart in a three-day ceremony on the Pacific coast, documented by his friend, the experimental filmmaker Stan Brakhage.

Hart and Broughton had two children, and built a counter-culture community along with friends including Alan Watts, Michael McClure, Anna Halprin, and Imogen Cunningham.

The 2012 documentary Big Joy: the Adventures of James Broughton was directed by Stephen Silha, Eric Slade, and Dawn Logson, with cinematographer Ian Hinkle.