In recording their fifth studio album, White Crosses, Against Me! also recorded acoustic versions of songs written for recording the album. In a press release on their website, Against Me! announced the single for release following their European tour, "to tide UK fans over until the band return for their headline shows with Fucked Up, Japanese Voyeurs and Crazy Arm in November," with an acoustic version of the song. In interviews, Grace has explained the meaning of the song: "I wrote the majority of the record in St Augustine, in Florida where I recently moved to and just around the corner from my house there's this church and the front lawn is covered in these little white crosses, and behind them there's a piece of text saying each of the crosses is there to represent how many abortions occur in America every day. So the song is in essence a pro-choice song, and seeing those crosses everyday, I fantasised about smashing them down, or driving my car through them. I think there's something to be said for acts like that - it'd be very gratifying, but instead I play music and write about this stuff. That's why I say in the lyric, "I wanna smash them all".