Caithlin De Marrais, Kaia Fischer and William Kuehn formed the band in late summer of 1995 and named it after the German-language poet Rainer Maria Rilke.

[3] In its earlier days, the band had a dual male and female vocal line-up; later, De Marrais would become the lead vocalist in a majority of their songs.

[citation needed] The band's many tours and intimate live shows at venues such as Brooklyn's North Six (which lead singer Caithlin De Marrais referred to as "home" during her final show because it also served as the band's rehearsal space), Washington, D.C.'s Black Cat, the Bowery Ballroom in NYC, and Chapel Hill's Cat's Cradle helped to grow its fan base and fuel album sales.

Referring to the intensity of the show and the enthusiasm of the crowd, Fischer at one point remarked jokingly in-between songs "we should break up every night."

[8] On February 29, 2024, the band announced via their Instagram that they would be playing at the Best Friends Forever festival in Las Vegas, marking their first show in 6 years.