Jagger and Richards recorded the song with their own group, the Rolling Stones, in 1965, for their fourth studio album Aftermath (1966).

Written at demand from their manager Andrew Loog Oldham, the song features several key characteristics of their music from this time, including a memorable tune and (arguably) misogynistic lyrics.

[1] This was primarily due to the group relying on recording cover versions, as opposed to writing original material.

[4] In the UK, it did not become the expected hit single, instead, the song entered Record Retailer on 27 April at a position of 47, before peaking at number 31 on 11 May.

[8] In a blind date for Melody Maker, Crispian St. Peters stated that "Take It or Leave It" didn't appear to have a middle eight.

[9] For New Musical Express, Derek Johnson calls the song a "good adaptation of a Rolling Stones composition.

"[10] He calls the song "unusual", noting the "fractually mid-tempo" and also praises the harmonies between Mike Pender and Frank Allen.

[18] This can in large part be attributed to the lax atmosphere in the studio;[19] according to Keith Richards, the album was not a result of a rush-job, and instead the band had a chance to relax during recording.

[20] The song had the Rolling Stones experimenting with instruments;[18] on the track, Mick Jagger sings, Richards is on acoustic guitar and backing vocals, Bill Wyman plays bass while Charlie Watts is on drums and finger cymbals.

[21] However, Gary Jucha writes that Jones was also responsible for the harpsichord part and also played percussion on the song.

However, by contrast to their extremely misogynistic "Under My Thumb", this song may be considered rather as a fair and "friendly" representation of hurt feelings of a suitor who has offered "just my life".

[33] On that album, "Take It or Leave It" is sequenced as the fourth track on side two, between "Mother's Little Helper" and "Ride On, Baby".