He joined the Joe Haymes orchestra in 1934, then played with Benny Goodman in 1936 and Artie Shaw in 1937. From 1937 to 1942, he worked and recorded with the bands of Red Norvo, Bob Crosby, Glenn Miller, Mildred Bailey, Frank Sinatra, Helen Ward, Judy Garland, Tommy Dorsey, and Ella Fitzgerald.
When World War II broke out, Zarchy was the first musician chosen by Glenn Miller for what became Miller's Army Air Force Band (officially, the 418th Army Band) where Zarchy played lead trumpet and was Master (First) Sergeant from 1942 to 1945.
After the war, singer Frank Sinatra invited Zarchy to move to Los Angeles, where he became a first-call studio musician. He played on the recordings of hundreds of vocalists, including Louis Armstrong, Tony Bennett, Dinah Shore, and The Mills Brothers. His trumpet is heard in the soundtracks of many classic Hollywood movies, including West Side Story (1961), Dr. Zhivago (1965) and the The Glenn Miller Story (1954).
During the 1960s and '70s, he played in the house bands of several CBS TV variety shows, including The Smothers Brothers Comedy Hour, The Danny Kaye Show and The Jonathan Winters Show, and was a member of the NBC Staff Orchestras in New York and Los Angeles.
In his later years, Zarchy made many music tours of Europe, South America, and Australia, as well as 32 concert trips to Japan. He tutored several young trumpet players who became successful performers and studio musicians. He died on April 12, 2009 at the age of 93.