Allmusic awarded the album with 4 stars and its review by Scott Yanow states: "This 1980 recording is an excellent example of David Sanborn's music. The highly influential altoist is joined by familiar studio veterans (including guitarist Hiram Bullock and drummer Steve Gadd) with bassist/composer Marcus Miller being a key figure in creating the funky rhythms and colorful backgrounds."
The recording won the 1982 Grammy Award for Best R&B Instrumental Performance for the track "All I Need is You". This was the first of seven Grammy Awards that have been achieved by the saxophonist to date. Buzz Feiten, riding high with the Larsen-Feiten Band at the time, adds his distinctive guitar tone to Let's Just Say Goodbye.