Victim of Love received widespread acclaim from critics. At Metacritic, which assigns a normalized rating out of 100 to reviews from mainstream publications, the album received an average score of 81, based on 20 reviews. Thom Jurek of AllMusic said, "Victim of Love showcases growth—and a sound not heard before on Daptone—while not straying from the gritty soul that established the singer; it is every bit as strong as its predecessor and more diverse." Jennifer Kelly of Blurt said, "Bradley and his band are such great interpreters and expanders of the soul tradition that you don't mind the nagging feeling that you've heard these cuts before." Michael Madden of Consequence of Sound said, "It's pretty uniformly vintage-soul stuff—barreling horns, wafting backing vocals, single guitar chords on the upbeat—but it never sounds antiquated." Ryan B. Patrick of Exclaim! said, "Victim of Love is meant to be taken literally; it's a rare and continued opportunity for a sexagenarian to finally get his chance in the soulful sun. Something the album proves that he's both appreciative of and not taking lightly." Alexis Petridis of The Guardian said, "Its quality never falters over the album's 40 minutes, from "Let Love Stand a Chance"'s gorgeous swoon to "Where Do We Go from Here"'s tense funk."
Nate Chinen of The New York Times said, "The new album is a little less pointed [than his debut], and a good deal less surprising. But Mr. Bradley, once again wailing against the convincing grit of Menahan Street Band, sounds bolstered by all the touring he has done over the last two years." James Reed of The Boston Globe said, "You hear him at the peak of his powers on the title track, whose acoustic soul reels in the band and lets Bradley tell his story, one wounded sentiment at a time." Andrew Burgess of musicOMH said, "He's lived it long and hard, and with this album Bradley continues to lay out all the goodness and badness of life and love, with soul to spare." Ryan Bort of Paste said, "If you were a fan of No Time for Dreaming you're going to be a fan of Victim of Love, and you shouldn't really need to know anything about it other than it's an album full of Charles Bradley songs." Dean Van Nguyen of Pitchfork said, "Victim of Love is ultimately a less successful record than No Time for Dreaming. For one, Bradley seems less connected with this set." Brent Faulkner of PopMatters said, "Victim of Love is best described as classic and authentic without feeling anachronistic in the least." Jody Rosen of Rolling Stone said, "It's Bradley's voice that seals the deal." Laura Studarus of Under the Radar said, "An album for the ages."