Almost immediately after recording "Puss 'n' Boots", Brad Roberts, Ellen Reid and Dan Roberts proceeded to prepare recording the follow-up, with Roberts acknowledging he went "through this phase where I was extremely prolific and I wrote like three records worth of material in a space of months."
The band pursued a decisively raw, acoustic, stripped-down sound for "Songs of the Unforgiven", and turned to the Internet in search of a recording studio that featured a pipe organ. By Googling "recording studios" plus “pipe organ”, the first result revealed Sacred Heart Studios in Duluth, Minnesota: a church originally constructed in 1894 that was later abandoned and eventually restored and turned into a venue for concerts. Having discovered Sacred Heart Recording Studio included a 1493-pipe, Felgemaker pipe-organ, the band drove to Duluth and recorded the album the following January.
The album is decidedly apocalyptic thematically, featuring lyrics replete with Biblical references and imagery. Other unconventional instruments appear on the record, including the harp and kettle drums.
Suzzy Roche (of female vocal group "The Roches" fame) provides backing vocals on the album's three sonnets.
The album resulted in the band receiving their best reviews since God Shuffled His Feet. Allmusic writer Rob Theakston gave the album 4 out of 5 stars and states that "with the Dummies stripped of all the electronic experimentation and quirky, Shel Silverstein-esque lyrics that hindered their past few releases, the haunting, sparse acoustic instrumentation makes a welcome return to the forefront of the songwriting process. This only reinforces and complements Roberts' existentially bereft baritone folk tales to their fullest potential. The sonnets used for lyrics are some of the Crash Test Dummies' darkest and most brooding in their entire catalog, talking tales of murder, sinister deeds, punishment, sin, and death. Songs of the Unforgiven is the record die-hard fans have been patiently waiting for, and it's outstanding.". In addition, Darryl Sterdan of the Winnipeg Sun, who also gave the album 4 out of 5 stars, states that the "over swirling pump organs, slowly strummed guitars and molasses-slow grooves, the baritone Roberts sombrely intones psalms of death, darkness and delivery, coming off like a cross between Leonard Cohen, Nick Cave, Tom Waits and Elmer Gantry. And the brooding neo-classical folk hymns of Songs of the Unforgiven come off as his most sincere, spiritual thoughts to date.".