The album was the first official release by Blake Schwarzenbach since the breakup of Jawbreaker two years earlier, marking a departure from punk rock he was known for in favour of a more indie rock sound. Although the new sound was met by mixed reception from Jawbreaker fans, Orange Rhyming Dictionary generated a new dedicated fan base for Jets to Brazil from both Jawbreaker and non-Jawbreaker fans alike.
Guitarist Brian Maryansky did not appear on the recording of the album, joining the band soon after the album was recorded. The final track from Orange Rhyming Dictionary, "Sweet Avenue" is often mistakenly attributed to the singer Damien Rice due to regular misattribution on peer-to-peer file sharing networks.
The name of the album is a play on the fact that, in the English language, there are no words that perfectly rhyme with "orange".