Summer had made her name the previous decade as the most successful female artist of the disco genre, releasing a vast selection of hit singles and albums on Casablanca Records. During this period however, Summer had felt that the label had exploited her and made her portray a sexually orientated image ("The First Lady of Love") with which she never felt comfortable. The label had also taken over other elements of Summer's personal life, to the point where she felt she had no control over her life or career. Having come out of a period of depression and rediscovering her Christian faith, Summer had made the decision to break away from Casablanca and file a lawsuit against them. After the lawsuit was eventually settled, Summer became the first artist to be signed to the newly established Geffen Records.
By 1980, banners reading "disco sucks" were seen everywhere and disco records became flops. Summer decided to leave the disco sound behind. The album was co-written and produced by Giorgio Moroder and Pete Bellotte, who had produced the vast majority of Summer's hits since their partnership with her began in 1974. Production for The Wanderer was rushed, Geffen wanted to get new product out because of Casablanca's plans to release Walk Away, another greatest hits collection. "We would have liked to do more tweaking, and have more time for production. But we just had to let it go," said Harold Faltermeyer about the recording of the album.
The album peaked at number 13 on the Billboard Album Chart - selling 600,000 US copies - and the title track hit number 3 on the US singles chart. However, two follow-up singles - "Cold Love" and "Who Do You Think You're Foolin'" - barely reached the Top 40. The album and its singles attained limited success on the UK charts. None of the singles cracked the UK Top 40.