The first album following their 1988 breakthrough The Trinity Session, The Caution Horses features a more conventional, polished sound than that album's spare, haunting country blues. As a consequence, the album was savaged by music critics, who charged that the band had sacrificed their distinctive style.
However, the album's accessibility to mainstream pop and rock audiences also meant that it received more radio airplay and spawned bigger chart hits ("Sun Comes Up, It's Tuesday Morning" and "Rock and Bird", which featured Bruce Hornsby on the piano on the single version) than The Trinity Session had. A number of critics have also noted that while the album is undeniably a disappointment when compared to its predecessor, when taken on its own merits it is a better album than its initial reception would suggest.
The band decided to record in a live setting as they had on The Trinity Session. This time they chose the Sharon Temple, a National Historic Site north of Toronto, as their studio. The recording proved challenging, as they had to adapt to the unique acoustics of the Sharon Temple and also the cool temperatures of a Canadian spring (they were not allowed to bring heaters into this unheated building due to its heritage designations). The cover of the album shows the band outside of this historic building.