Levi was born in Giessen, Germany, the son of a rabbi. He was educated at Giessen and Mannheim, and came to Vinzenz Lachner's notice. From 1855 to 1858 Levi studied at the Leipzig Conservatory, and after a series of travels which took him to Paris, he obtained his first post as music director at Saarbrücken, which post he exchanged for that at Mannheim in 1861. From 1862 to 1864 he was chief conductor of the German Opera in Rotterdam, then till 1872 at Karlsruhe, when he went to Munich, a post he held until 1896, when ill-health compelled him to resign. Levi also taught at the Leipzig Conservatory, where his pupils included the conductor Emil Steinbach.
Levi's name is indissolubly connected with the increased public appreciation of Wagner's music. He was a longtime friend of Wagner; when preparing for the inaugural Bayreuth Festival, he wrote to his father, "Wagner is the best and noblest of men ... I thank God daily for the privilege to be close to such a man. It is the most beautiful experience of my life". He conducted the first performance of Parsifal at Bayreuth in 1882, and was connected with the musical life of that place during the remainder of his career. He visited London in 1895, and died in Munich in 1900 and was interred in a Mausoleum in the grounds of his villa later that year in Garmisch-Partenkirchen.