Varga was active in the DISZ, student organisation of the Hungarian Working People's Party, within which he was regarded as an intellectual. He was part of a group which in 1954 founded the Petofi discussion group, in opposition to Mátyás Rákosi. When it was revived in 1956, he became Deputy Secretary of its Provisional Bureau. The group, while influential, attempted to quell the Hungarian Revolution, fearing a violent backlash.
Following the defeat of the revolution, Varga escaped to Austria, then settled in Paris, where, following discussions with Pierre Broué, he became a Trotskyist and founded the League of Revolutionary Socialists of Hungary with other exiles. He joined the Internationalist Communist Organisation (OCI) in 1962, and in 1963 his league affiliated to the International Committee of the Fourth International (ICFI).
In 1972, the OCI decided to leave the ICFI and instead set up the Organising Committee for the Reconstruction of the Fourth International. Varga initially sided with the OCI, but soon after came into conflict with them and was expelled. The following year, he founded the International League for the Reconstruction of the Fourth International. The OCI claimed that he had been both a CIA and a KGB agent, but an international committee consisting of representatives from Lutte Ouvrière, the Ligue Communiste Revolutionnaire, the Socialist Workers Party (US) and the International Spartacist Tendency found no proof of this.
In 1984, Varga was expelled by an opposition group within his own international, and instead formed the Group of Opposition and Continuity of the Fourth International. In 1990, this dissolved itself into the Workers International to Rebuild the Fourth International, of which Varga became Secretary.
- "The Hungarian Revolution 1956 - Part 1", Permanent Revolution, October 2006
- The Present Day Relevance of the Transitional Programme, Revolutionary History Vol.3 No.1
- Robert Jackson Alexander, International Trotskyism, 1929-1985