Kallman was born in Brooklyn of Jewish ancestry. He received his B.A. at Brooklyn College and his M.A. at the University of Michigan. He published three collections of poems, Storm at Castelfranco (1956), Absent and Present (1963), and The Sense of Occasion (1971). He lived most of his adult life in New York, spending his summers in Italy from 1948 through 1957 and in Austria from 1958 through 1974. In 1963 he moved his winter home from New York to Athens, Greece, and died there at the age of 54. Kallman had been the beneficiary of the entirety of Auden's estate, but himself died intestate, with the result that the estate was inherited by his next-of-kin, his father, a New York dentist in his eighties.
Together with his lifelong friend (and sometime lover) W. H. Auden, Kallman wrote the libretto for Stravinsky's The Rake's Progress (1951). They also collaborated on two librettos for Henze, Elegy for Young Lovers (1961) and The Bassarids (1966), and on the libretto of Love's Labour's Lost (based on Shakespeare's play) for Nicolas Nabokov (1973). They also wrote a libretto "Delia, or, A Masque of Night" (1953), intended for Stravinsky, but never set to music. They were commissioned to write the lyrics for Man of La Mancha, but Kallman did no work on the project, and the producers decided against using Auden's contributions.
He and Auden collaborated on a number of libretto translations, notably The Magic Flute (1956) and Don Giovanni (1961). Kallman also translated Verdi's Falstaff (1954), Monteverdi's The Coronation of Poppea (1954) and many other operas.