Baller was born in 1909 in Brody, Galicia, then part of Poland (now Ukraine). At age 8, he went to Vienna to study piano with a former student of Franz Liszt and at 13 he performed with the Vienna Philharmonic. In March 1938, Nazi soldiers learned that he was a pianist and a Jew, and arrested him, beat him and crushed his hands. Baller's fiancée, Edith Strauss-Neustadt, interceded on his behalf with the Polish Consul in Vienna and helped to restore his hands so that he could resume his career. The couple escaped to Budapest, where they were married before coming to the United States in 1938.
The Ballers met violinist Yehudi Menuhin and cellist Gabor Rejto in New York, and beginning in 1941 they lived with their daughter, Nina, at Menuhin's Alma Estate in Los Gatos, California. Later on, he became the teacher of pianist Jerome Rose. Under Yehudi Menuhin's patronage, Baller, Gabor Rejto and Roman Totenberg formed the Alma Trio in 1942-1943 at Menuhin's Alma estate in Los Gatos, California. For several years Baller accompanied Menuhin in performances throughout the world and performed in chamber concerts. Baller died at his home in Palo Alto in 1994. Violinist Maurice Wilk joined the Alma Trio after Totenberg retired. Following Wilk's sudden death in 1963, violinist Andor Toth joined the Alma Trio and remained their violinist for the next 15 years.