Diamonds in the Rough was recorded at Atlantic Recording Studios in New York City. The album's sound honed in on the Appalachian "high lonesome" influences evident on Prine's eponymous debut LP and its bluegrass instrumentation reflects Prine's fascination with early American folk and country music. The album was produced by Arif Mardin and features Prine's brother Dave on dobro, banjo, and fiddle while Steve Goodman also contributes on guitar and harmonies. In the Great Days: The John Prine Anthology liner notes, Prine tells David Fricke that the album was recorded in three days at a cost of $7,200, "including beer," and adds "I just wanted to do Diamonds the way I was used to playing music at my house with Dave and Steve. But it's taken me years to figure out how to balance those first two records. About every other record, after making a real studio or rock 'n' roll album, I'd come back and do a Diamonds in the Rough."
The material on Diamonds in the Rough strike a musical balance between infectious up-tempo hootenannies and stark, allegorical compositions with recitations that recall Hank Williams' recordings as Luke the Drifter. In an interview with Paul Zollo for Bluerailroad magazine, Prine stated that "Sour Grapes" and "The Frying Pan" were two of his earliest songwriting efforts, explaining that he "had a girlfriend whose father was a janitor. And the reason I’m telling you that is because he had access to a tape recorder, and nobody else I knew had one. They were really rare. A reel-to-reel. He got it from the language department. It was broken and he fixed it and had it at home. And I sat down and taped three songs for this girl and her sister. And the three songs were 'Frying Pan', 'Sour Grapes', and 'Twist & Shout'...Years later, I ended up marrying that girl. She was my first wife. She found the tape. It was after I had made the first album, so I put two of those songs on Diamonds in the Rough. And those were the first songs I remember writing." Prine also added that he wrote "Souvenirs" in his car, "a 65 Chevelle. Driving to the Fifth Peg. Like the 5th or 6th time playing there. I used to play there just Thursdays after they hired me. They hired me from that open stage the very first time I sang for the crowd. They invited me back a week later, and I did it again for an open stage...So about the fifth time I was driving down there I thought, God, the same people are gonna be sitting there. I better have a new song. So I wrote 'Souvenirs' in the car on the way down. And then I thought I’d come up with a melody. And I thought I had come up with a pretty sophisticated melody in my head, and I was surprised to find out it had the same three chords that all my other songs have. Really surprised. I thought I had written a jazz melody." Prine often performed "Souvenirs" with his friend and fellow songwriter Steve Goodman, who played on the original recording, and in the same interview the singer confessed, "Yeah, I can still hear him playing it. He played a back melody, so that you could barely hear the difference of who was playing. On tape or when we did it live. And I realized a large part of what he was doing was making it sound like I was playing the good part. And that’s basically the kind of guy he was."
Prine admitted to Bluerailroad that he borrowed the melody for "The Late John Garfield Blues" from the Jimmie Rodgers song "Treasures Untold", stating "It's a really pretty ballad that he wrote. I learned that song early on and I always wanted to use that G to the B7." The title track, which was originally recorded by the Carter Family, is sung by Prine, his brother Dave, and Goodman without accompaniment. Regarding "Yes, I Guess They Oughta Name A Drink After You", the John Prine Shrine website quotes the singer: "I was going for a Hank Williams kind of song. Steve Goodman always told me that if I'd taken another couple of minutes and put a chorus to the song - there isn't any, just a tag line to every verse - that it would have been a hit country song. And I was set in my ways. Once a song was done, it was done." Two songs on the album, "Take The Star Out of The Window" and the allegorical protest waltz "The Great Compromise", deal with America's involvement with the Vietnam War, with Prine saying of the latter in the Great Days anthology, "The idea I had in mind was that America was this girl you used to take to the drive-in movies. And then when you went to get some popcorn, she turned around and screwed some guy in a foreign sports car. I really love America. I just don't know how to get there anymore."
Music critic Robert Christgau deemed Diamonds in the Rough "Not as rich as the debut, but more artlessly and confidently sung—the gruff monotone avoids melodrama in favor of Prine's own version of good-old-boy..." Critic William Ruhlman of AllMusic opines, "John Prine's second album was a cut below his first, only because the debut was a classic and the follow-up was merely terrific...Diamonds in the Rough demonstrated that Prine had an enduring talent that wasn't exhausted by one great album." In 1993 David Fricke wrote, "It was, in essence, John Prine-in-the-rough, a superb collection of tunes and tales recorded with acoustic, no-frills living room elegance." Steven Stolder of Amazon.com writes that the "fireworks that accompanied the appearance of John Prine's 1971 debut cast a long shadow over its deceptively modest follow-up" but argued that its strength "lies less in the tunes themselves than in how sympathetically they're performed and sequenced."