The song features LL Cool J vacillating about moving to California, rapping in the chorus, "I'm going back to Cali", followed by "I don't think so". In the verses, he describes a trip to Los Angeles in which he meets several women but is scared off because they are too sexually aggressive. The basic concept for the song was Rubin's, based on his own ambivalence about possibly moving from New York City to Los Angeles. (Rubin moved to LA later in 1988.)
The 1990 music video "Going Back to Brooklyn", starring Colin Quinn and directed by Ben Stiller, parodied the song and music video. In the video, Quinn raps about being afraid to return to his home town because he is no longer tough enough.
The 1990 Sonic Youth song "Kool Thing" includes references to "Going Back to Cali" in both the song's lyrics and its music video; the song was intended as an attack on LL Cool J for his perceived misogyny.
The music video for Eminem's 2013 song "Berzerk" referenced the "Going Back to Cali" video in a scene showing Rick Rubin playing cards.
In 2015, Jimmy Fallon cold-opened an episode of The Tonight Show, with a shortened version of the song and music video, to kick off a week of hosting the show in California. LL Cool J has a cameo appearance in the video.