Vocalist Ronnie James Dio was first introduced to Black Sabbath guitarist Tony Iommi in 1979 by Sharon Arden, ironically the future wife of the band's recently-fired vocalist Ozzy Osbourne. Initially Dio and Iommi discussed forming a new band, not a continuation of Black Sabbath. The pair met again by chance at The Rainbow on Sunset Strip in Los Angeles later that year. Both men were in similar situations, as Dio was seeking a new project and Iommi required a vocalist. Said Dio of the encounter, "It must have been fate, because we connected so instantly." The pair kept in touch via telephone, until Dio arrived at Iommi's Los Angeles house for a relaxed, getting-to-know-you jam session. On that first day the duo wrote the song "Children of the Sea".
Black Sabbath's line-up was in chaos prior to the recording of Heaven and Hell. Not only had Osbourne recently been replaced, but drummer Bill Ward was battling personal issues that would see him eventually leave the band. Original demo recordings for the album featured Geoff Nicholls on bass, as the band's longtime bassist Geezer Butler was going through a divorce and his future with the band was in question. In fact, when Dio first joined the band he was doubling as bassist and vocalist, having played bass in the band Elf in the early 1970s. At one point Iommi contacted close friend Frank Zappa for help finding a bassist. Zappa offered his bassist for the Heaven and Hell sessions but Iommi preferred a permanent member. Eventually, Butler returned and Nicholls stayed on as the band's unofficial keyboardist. Former Elf and Rainbow bassist Craig Gruber also rehearsed with the band during this period, though the true extent of Gruber's involvement is unclear. In a 1996 interview, Iommi stated that Gruber rehearsed with the band only "for a bit". Gruber has stated that his contribution was much more substantial; he claims to have co-written most of Heaven and Hell's songs and that it was actually he and not Butler who played bass on the album. Despite not being credited for his contributions, Gruber claims "we came to a suitable financial arrangement". Iommi later stated in his 2011 autobiography that Gruber did in fact record all the bass parts on the Heaven and Hell album, but that Butler re-recorded the parts upon his return to the band.
Dio suggested the band hire producer Martin Birch for the album. Birch was the first outside producer the band had used since 1971's Master of Reality, as Iommi had primarily produced the band's albums since that point on his own. Iommi stated that the band felt during recording that they were creating "something very special". The album title Heaven and Hell was chosen from a lyric written by Dio about the highs and lows of being a musician.
Drummer Ward has stated that he has "no memory" of making the album, due in large part to his alcoholism. His behaviour became quite erratic; when the band began touring in support of Heaven And Hell, Ward began dictating long and rambling press releases to the band's public relations representatives after every show, instructing them to "get that out on the news wires tonight". Ward's personal issues, which included the deaths of both his parents, would soon force him to leave the band. Dio reportedly answered the telephone in his hotel room one morning mid-tour to hear Ward say "I'm off then, Ron", to which Dio replied "That's nice Bill, where are you going?". "No, I'm off mate. I’m at the airport now...", indicating that he was incapable of completing the tour with the band. American drummer Vinny Appice was quickly brought in to replace him.
The album was successful, becoming their highest-charting album (No. 9 UK, No. 28 US) since 1975's Sabotage and the third highest-selling album of Black Sabbath behind Paranoid and Master of Reality, respectively.
In the UK, it became the third Black Sabbath studio album to attain silver certification (60,000 units sold) by the British Phonographic Industry, achieving this in November 1980. It subsequently attained gold certification (100,000 units sold) in April 1982, the only Black Sabbath studio album to be thus certified.
The album's cover art was taken from a painting by artist Lynn Curlee, Smoking Angels, inspired by a 1928 photograph of women dressed as angels smoking backstage during a break at a college pageant. Curlee was also commissioned to do an album cover for Blue Öyster Cult by Sandy Pearlman, who managed both bands.
The album's back cover illustration of the band was drawn by artist Harry Carmean.
"Heaven and Hell" is briefly covered acoustically by Tenacious D for Dio's "Push" music video. At the beginning of the video, Jack Black and Kyle Gass are shown standing on a sidewalk busking and singing "Heaven and Hell" with a few alternate lyrics. They stop when Dio walks up to them and tells them that he'll pay them if they would just play some Tenacious D songs.