Born in London, he was educated at Wagner’s and at Eton College. Jaffé's undergraduate studies were delayed for four years by World War II, during which time he served in the RNVR. He came up to King's College, Cambridge in 1945, studying History before changing to English, in which subject he got a First. He became President of the Marlowe Society, and was editor of Granta while a student. After Cambridge, he studied art history at the Courtauld Institute, where he attended Johannes Wilde’s lectures and had access to the Seilern Collection; this was followed by research at Harvard on Rubens and his contemporaries.
He became a Fellow of King's in 1952, holding the position until his death, and was appointed as the University's only Assistant Lecturer in Fine Arts in 1956, and began undergraduate teaching in the subject. He held the post for four years until going to Washington University in 1960, where he was briefly Professor of Renaissance Art until returning to Cambridge University in 1961, when he was appointed Lecturer in Fine Arts. In 1968 he was appointed Reader in History of Western Art, and he became Head of Department of History of Art in 1970, a position he held until 1973 and his appointment to the Directorship of the Fitzwilliam Museum.
Jaffé married Patricia Milne-Henderson in 1964, and they had two sons and two daughters. Jaffé owned the country house Clifton Maybank near Yeovil in Somerset. He was appointed a CBE in 1989. He died on 13 July 1997. A bronze portrait bust of Jaffé by Elisabeth Frink is in the Fitzwilliam.
- Van Dyck's Antwerp Sketchbook 1966
- Rubens 1967
- Jacob Jordaens 1593–1678 1968
- Rubens and Italy 1977
- Rubens: catalogo completo 1989
- Old master drawings from Chatsworth 1993
- editor of The Devonshire Collection of Italian Drawings 1994