Jaffe was born in Harlem to Russian Jewish immigrants and began his career as an office boy for Paramount-Famous Players-Lasky Company where he was brother-in-law to B.P. Schulberg was an executive which no doubt helped him get his first job at Paramount. He eventually worked his way up through the ranks to become the executive in charge of production. In the early 1930s he worked at Columbia Pictures briefly before leaving to join the talent agency founded by his older sister, Adeline Jaffe Schulberg. When she opened a branch in London, he assumed control of the agency. He successfully represented several stars of the era, including Lauren Bacall, Peter Lorre, Humphrey Bogart, Fredric March, David Niven, Zero Mostel, Richard Burton, and Stanley Kubrick, until the 1950s when his business was negatively affected by investigations of many of his clients by the House Un-American Activities Committee investigations into Hollywood.
In 1959, he moved to London and became a film producer and a well known collector of modern art.