Featuring Dolphy in a quintet with trumpeter Freddie Hubbard, vibraphonist Bobby Hutcherson, bassist Richard Davis and drummer Tony Williams, it was generally considered by critics as one of the finest albums issued on Blue Note, and widely viewed as one of the high points of 1960s avant-garde jazz.

[2] The album cover designed by Reid Miles features a photo of a "Will Be Back" sign displayed in a shop window showing a seven-handed clock.

Bobby Hutcherson had participated in the July 1963 recording sessions that produced the Dolphy albums Iron Man and Conversations.

Dolphy expressed his excitement regarding the opportunity to work with Hutcherson again, stating: "Bobby's vibes have a freer, more open sound than a piano.

"[3] Bassist Richard Davis had also participated in the July 1963 sessions, during which he recorded a number of duets with Dolphy ("Come Sunday", "Ode to Charlie Parker", and "Alone Together").

[8] The title of the album's first track, "Hat and Beard", refers to Thelonious Monk, about whom Dolphy stated: "He's so musical no matter what he's doing, even if he's just walking around.

[3] This track features the album's most conventional bebop-based theme; Dolphy noted that "Everybody holds to the construction for the first 13 bars, then - freedom.

[13] Writer Martin Williams stated that on Out to Lunch!, "we hear a full development of [Dolphy's] talent, in its five, finely crafted compositions, and in his equally well-conceived solos.

"[14] In a review for AllMusic, Steve Huey described the album as Dolphy's "magnum opus, an absolute pinnacle of avant-garde jazz in any form or era.

Its rhythmic complexity was perhaps unrivaled since Dave Brubeck's Time Out, and its five Dolphy originals... were a perfect balance of structured frameworks, carefully calibrated timbres, and generous individual freedom.

The organized mayhem starts with Dolphy's tunes, often featuring wide, wide leaps in the melody and ratchet-gear rhythms...in time you could hear its influence in Anthony Braxton's or Roscoe Mitchell's zigzag solos and odd timbres, in David Murray's yawping bass clarinet and Jason Adasiewicz's clanking vibes.

"was an effort to break our expectations about the very nature of jazz", and stated that "In addition to dispelling the notion that improvised solos should maintain the mood of the melody, Dolphy called for his sidemen to rethink their preconceived notions about improvisation", noting that "Dolphy and company re-examined the very framework of this music.

He praised Dolphy's bass clarinet playing on the opening track as "as complete a statement of musical independence—from fashion, commercial concerns, and tradition—as is likely to be found.