Guesting on the album would be Penetration's lead vocalist Pauline Murray, who sang a duet with Peter Perrett on the Johnny Duncan cover song "Fools", and also provided backing vocals for "Me and My Shadow".

The album marked the only time that the band had released a cover version with the aforementioned "Fools", which was later released as a single, and Baby's Got a Gun also marked the only time that a guitarist other than the band members Perrett and John Perry had been credited, with Barrie Evans providing rhythm guitar, in addition to playing percussion.

In his consumer guide for the Village Voice, Robert Christgau gave the album a B+ and commented that "Prepunk and for that matter prepub, Peter Perrett may well have been an only one and he fits in now only because this is such a tolerant and/or commercially desperate time.

In a retrospective review for AllMusic, critic Mark Deming wrote that "Baby's Got a Gun is clearly the weakest of the Only Ones' three original albums, but for all its faults there's plenty here that testifies to the band's strengths; "Why Don't You Kill Yourself", "Strange Mouth", and "The Big Sleep" are splendid songs that show the band still had the goods and "Trouble in the World" and "The Happy Pilgrim" confirm they could reach for a poppier sound without losing their personality in the process."

All tracks are written by Peter Perrett, except where notedCredits are adapted from Baby's Got a Gun liner notes.