After the lack of commercial success with their previous album Even Serpents Shine, the Only Ones decided that it was time for a slight change of pace in their career. In an attempt to give their music a more contemporary sound, they joined forces with Colin Thurston, an in-demand producer famed for having co-engineered David Bowie's "Heroes" and Iggy Pop's Lust For Life (both 1977). Guesting on the album would be Penetration's lead singer, Pauline Murray, who provided co-lead vocals for "Fools" and backing vocals for "Me and My Shadow".
The album was re-released in Europe in 2009 on Sony Music Entertainment, featuring rare bonus content. The reissue was a CD which comprises 15-tracks. It includes the original album digitally remastered from the original 1/2" mix tapes; alongside three bonus tracks.
In his consumer guide for The Village Voice, Robert Christgau gave the album a B+ and commented that "Prepunk and for that matter prepub, Peter Perrett may well have been an only one and he fits in now only because this is such a tolerant and/or commercially desperate time. He's not "power pop," of course--on record, at least, power's got nothing to do with it."
In a retrospective review for AllMusic, critic Mark Deming gave the album three out of five stars and wrote that "Baby's Got a Gun is clearly the weakest of the Only Ones' three original albums, but for all its faults there's plenty here that testifies to the band's strengths; "Why Don't You Kill Yourself," "Strange Mouth," and "The Big Sleep" are splendid songs that show the band still had the goods and "Trouble in the World" and "The Happy Pilgrim" confirm they could reach for a poppier sound without losing their personality in the process."