In 1963 Waller adopted the name Mickey Finn – after having heard about the drummer named Micky Waller – and joined with John "Fluff" Cooke (keyboards), John Burkett (bass), Alan Marks (lead vocals) and Richard Brand (drums) to form "Mickey Finn & the Blue Men", who released their debut single in January 1964.

Reverting to his original name (percussionist Mickey Finn had by then become famous as a member of T.Rex), Waller played guitar with Sam Gopal (1969–70) and the Heavy Metal Kids (1972–74), before relocating to France and becoming a sought-after session musician, notably with Nino Ferrer.

As Mickey Finn, he returned to the UK and joined Steve Marriott's All Stars in July 1975, and in May 1976 Finn and fellow All Star Greg Ridley (bass) formed The Fallen Angels with Twink (drums), Guy Humphries (guitar) and (initially) Bob Weston (guitar) and Keith Boyce (drums).

The band crashed their van on the way to their first gig, and Finn was the only remaining original member when they re-emerged in July 1977 backing ex-Pretty Things singer Phil May.

By 1999, Mickey Finn & the Blue Men had reformed and recorded the Black Hole album, followed by the Go Clean EP (Ten Minutes Productions, June 2004).