He has played piano since the age of six. He studied piano and composition at the conservatory in Bratislava. He left the conservatory after three years to become a member of the group Prúdy, and contributed to the legendary album Zvoňte zvonky. He left Prúdy as suddenly as he had left conservatory only to establish the first Czechoslovak art rock band Collegium Musicum.
The repertoire of Collegium Musicum, comprising mostly instrumental pieces, included reinterpretations of the themes of classics such as Joseph Haydn, Béla Bartók and Igor Stravinsky, complemented by original compositions. Already at this stage his work bore signs of postmodernism (Eufónia of the album Konvergencie), which later became the basic principle of his work.
When Collegium Musicum disbanded in 1979, Varga started a solo career. Among other achievements, he became a pioneer of absolute improvisation (real-time composition) in Slovakia. In the meantime he continued to contribute to popular music. His ongoing collaboration with Pavol Hammel led to five successful albums and to the first rock musical in Slovakia.