Born in London, England, he reportedly began his study of the cornet with only its mouthpiece; his family could not afford the instrument itself. After immigrating to the United States, he began a significant musical career as a cornet soloist and was billed as The World's Greatest Cornetist. He was widely regarded as a foremost player, although the claim of World's Greatest has some challengers. He was a member of Patrick Gilmore's band for several years, performing with them at the Centennial Exposition in Philadelphia in 1876. He was also a tester and promoter for C.G. Conn, manufacturer of musical instruments.
His son, Jules Levy, Jr., was also an accomplished cornetist and trumpeter. He recorded from 1919 to 1932, often with groups led by Joseph Samuels. Levy was briefly married to actress Mary “Minnie” Conway in the 1870s. Their son, Frederick Conway Levy, was the popular stage and screen actor Conway Tearle.