Milner was born in Bristol, and educated at Douai School, Berkshire. He was awarded a bursary to attend the Royal College of Music, where he studied piano with Herbert Fryer and theory with R. O. Morris. He studied composition privately with Mátyás Seiber. Milner's own teaching career began at Morley College, London, where he taught music theory and history from 1948–64. He was Lecturer in Music at King's College London, from 1965–71, when he moved to Goldsmiths' College as Senior Lecturer, becoming Principal Lecturer in 1974. In 1980 he was appointed full-time Principal Lecturer at the Royal College, where he had taught part-time since 1961. He remained in this post until his retirement in 1989.
Milner had close academic ties with North America. Beginning in 1964, he gave frequent summer lecture tours in the USA and Canada. Milner's teaching interests centred on twentieth-century British music and on sacred and liturgical music. He was Composer-in-Residence at the Summer School of Liturgical Music at Loyola University New Orleans in 1965 and 1966, and the first Director of Spode Music Week, an annual residential Music school that places particular emphasis on the music of the Roman Catholic liturgy.
Milner's early compositions were influenced by Michael Tippett, who was Morley College's music director at the time of his appointment to the staff. As he developed his own voice, he continually sought new ways of expressing himself within an essentially tonal style. Contrapuntal rigour and the influence of plainsong are evident in most of his works.
A commission for the Leicestershire Schools Music Festival in 1967 resulted in a Te Deum, first performed by the Leicestershire Schools Symphony Orchestra and Chorus in May 1967 under the direction of the composer. From 1954-65 Milner was director and harpsichordist of the London Cantata Ensemble, with whom he gave the first broadcast performances in the UK of many Buxtehude cantatas as well as frequently conducting performances of his own music.
Milner was homosexual but non-practising due to his Catholic faith.
Milner was diagnosed with multiple sclerosis in his 40s. As the illness progressed, composition became an increasingly difficult activity. His last work, the Oboe Concerto, was completed in 1994. He spent the last two years of his life in Spain and died there on 22 September 2002.