Rehearsing My Choir is the fourth full length album by The Fiery Furnaces and was released in 2005. It is a concept album featuring Olga Sarantos, grandmother of the band's Matthew and Eleanor Friedberger, narrating stories relating to her life. Though the instrumentation is as complex as on the band's previous releases, the vocals mostly take the form of spoken word. At points Eleanor Friedberger speaks or sings words corresponding to the character of a younger Olga Sarantos, the two engaging in dialogue. This aspect has made many draw relations between the album and a radio play. Olga Sarantos died on December 31, 2007. 
Like Blueberry Boat, the album shifts around in a non time-linear way. It also includes a few sections separate from the main story. In the album's press release, Matthew Friedberger, the songwriter and lyricist, gives a summary and explanation:
Dear Listener, Tracks 3 and 4 take place in the 40's; tracks 5 and 6 in the 20's and 30's; track 7 in the later 50's; track 8 starts in the very early 40's; track 9 goes back and forth; track 10 takes place in the early 60's; the final track takes place in the early 90's. Track 2 takes place a few years ago; track 1 took place when it was recorded. The action depicted in "The Wayward Granddaughter" and "Slavin' Away" does not include the character Olga Sarantos plays on the rest of the record. "Slavin' Away" imagines that character—the main character—fantasizing, a bit remotely, about the hard lot of other women. Now, I wouldn't guess that the Main Character actually thought the woman concerned was riding around in a Norton side-car and operating her own cottage industry trinket assembly/sweatshop: but it might have pleased her to picture it so. "The Wayward Granddaughter" is about a different Greek-American grandmother and her popular granddaughter ("Connie"). They're from Chicago's south suburbs and don't figure in the rest of the record; I wanted to have another (slightly younger) grandmother and family in there for perspective or comparison's sake, so to speak. Thank you for your time, Matthew Friedberger
Like much of the band's output, the album received both enthusiastically positive and negative reviews from music critics. JSpicer at Tiny Mix Tapes called it a "Great American Novel via guitars, drums, bells, and whistles" and admired its envelope-pushing sound and unique storytelling. Justin Cober-Lake at PopMatters found the storytelling "neither nuanced nor complex" and wrote that the music "lacks the focus, coherence, and development to be rewarding beyond a novelty listen."