Due to financial restraints, the early Sweet Billy Pilgrim recordings up until 2012's Crown and Treaty were (apart from the acoustic drums) produced in a garden shed using a laptop and one quite good microphone.
Released on their own label (Luxor Purchase) and featuring hand-stamped artwork, it contains three songs: "Stars Spill Out of Cups", "God in the Details", and "Atlantis".
Critical response was positive, with all major UK national papers awarding the record four or five stars out of five.
Besides their work as Sweet Billy Pilgrim, Bishop, Hamer, and Elsenburg have worked together as session musicians, having toured as members of Martin Grech's band during the Open Heart Zoo era, and playing on the Toyah Willcox album Velvet Lined Shell.
[11] Hamer toured with The Boy Least Likely To until 2008, and his drumming is featured on the James Dean Bradfield album The Great Western.
Due to the complexity and layering of sounds used on the recordings, recreating the songs live requires the members to play many different instruments on stage.
On 13 July 2010, their song "Future Perfect Tense" was made available as a download for the game Rock Band 2.
In April 2013, in collaboration with Mojo magazine the band gave away a free download of Crown and Treaty with the full artwork in exchange for signing up to their mailing list.