Kirby's next album as the Caretaker, Theoretically Pure Anterograde Amnesia (2005) would abandon the haunted ballroom concept and install themes of memory loss.

[5] Kirby's next release after We'll All Go Riding on a Rainbow, Theoretically Pure Anterograde Amnesia (2005) would add several layers of complexity to the pseudonym, as it directly explored memory loss.

AllMusic critic John Bush stated the record could even convince listeners that "Kirby has actually spent time in a haunted ballroom".

[8] Writing for Stylus Magazine, Todd Burns criticized the 50-minute length, which made it fall "prey to the problem of most screwed music".

[10] Noting the record's difference to Kirby's V/Vm releases, Andy Slocombe of Comes with a Smile felt the album's "depth is sometimes breathtaking, and the overall effect is totally all-consuming," and added that it is "highly recommended, if you're up to it.

"[9] When reviewing the Caretaker's later record Patience (After Sebald) (2012), Fact argued We'll All Go Riding on a Rainbow was when the haunted ballroom concept started leaving listeners "wondering just how much mileage could be left in such an idiosyncratic and specific aesthetic.