The song features LL Cool J vacillating about moving to California, rapping in the chorus, "I'm going back to Cali", followed by "I don't think so". In the verses, he describes a trip to Los Angeles in which he meets several women but is scared off because they are too sexually aggressive. The basic concept for the song was Rubin's, based on his own ambivalence about possibly moving from New York City to Los Angeles.
The 1990 music video "Going Back to Brooklyn", starring Colin Quinn and directed by Ben Stiller, parodied the song and music video. In the video, Quinn raps about being afraid to return to his home town because he is no longer tough enough.
The 1990 Sonic Youth song "Kool Thing" includes references to "Going Back to Cali" in both the song's lyrics and its music video; the song was intended as an attack on LL Cool J for his perceived misogyny.
The music video for Eminem's 2013 song "Berzerk" referenced the "Going Back to Cali" video in a scene showing Rick Rubin playing cards.
In 2015, Jimmy Fallon cold-opened an episode of The Tonight Show, with a shortened version of the song and music video, to kick off a week of hosting the show in California. LL Cool J has a cameo appearance in the video.
The song was used in the third season of the Netflix series Narcos. The final episode of the season was named Going Back to Cali, a play on words between Cali (short for California) and Cali (the city in Colombia where the show takes place).