Milford was born in Oxford, son of Sir Humphrey Milford, publisher with Oxford University Press. He attended Rugby School from 1916 where his musical talent for the piano, flute and theory was recognised, and studied at the Royal College of Music from 1921 to 1926. His composition teachers were Gustav Holst and Ralph Vaughan Williams, and he studied harmony and counterpoint under R. O. Morris. He also studied organ.
In 1927, he married. Realising that he would not be able to make a living solely as a composer he worked for a time with the Aeolian Company correcting Duo-Art pianola rolls until 1930. He also taught part-time at Ludgrove School (where his pupils included the music enthusiast George Lascelles, later 7th Earl of Harewood) and at Downe House School. In 1929 he had met fellow-composer Gerald Finzi, with whom he found he had much in common, personally and musically, and the two formed a lifelong friendship.
His early compositions met with some success, his Double Fugue Op. 10 winning a Carnegie Prize and being performed by the London Symphony Orchestra under Ralph Vaughan Williams. In September 1931 his oratorio A Prophet in the Land Op. 21 was performed in Gloucester Cathedral as part of the Three Choirs Festival - the work was somewhat overshadowed by the splash made by William Walton's Belshazzar's Feast performed the same year. In 1937 a performance of his Concerto Grosso Op. 46 was directed by Malcolm Sargent, and his Violin Concerto Op. 47 was broadcast by the BBC in early 1938.
At the outbreak of the Second World War Milford volunteered for the army, and was posted to the Pioneer Corps. After just one week, he suffered a breakdown, and after treatment he and his family moved to Guernsey. His depression was deepened by the death of his mother in 1940. He returned to England, to teach and compose, but soon afterwards his five-year-old son, Barnaby, was killed in a road accident. His grief at this tragedy prompted him to attempt suicide; sent back to hospital after this, he tried to kill himself yet again while a patient there. In 1946, he had recovered sufficiently to resume teaching (at Badminton School) and to undertake musical activities. He continued composing throughout this period.
After the death of his father in 1952, he was prescribed occasional shock therapy. He did continue to enjoy successes: his Overture for a Celebration Op. 103 was performed under John Barbirolli at the 1955 Cheltenham Music Festival. He also continued to receive moral and material support from his friends, Finzi (who led a performance of Fishing by Moonlight Op 96 in 1956) and Vaughan Williams (who arranged a performance of the Concertino Op 106 in 1958, and gave financial help).
The deaths of Finzi (1956) and Vaughan Williams (1958) affected Milford deeply, aggravating the effects of his physical decline, which involved loss of vision and impaired balance. He died by his own hand, taking an overdose of aspirin in December 1959.
It has been observed that Milford's writing shows strongly the influence of Vaughan Williams, as might be expected. His use of diatonic melodies, often harmonised with gentle discords, and with false relations occurring occasionally, has led Erik Blom (1942) to crystallise these musical traits (also shown by other English composers of the period) as "musical Englishry". Vaughan Williams once wrote to Adrian Boult, "If I wanted to show the intelligent foreigner something worth doing which could only possibly come out of England, I think I would show him something of the work of Milford…" 
Despite the tragic events of Milford's life, and his resultant depression, he seems to have had a capacity for incidental enjoyment and his music is by no means gloomy. Indeed, a factor contributing to Milford's depression was that his brand of English music, as handed down from Vaughan Williams and Holst, was going out of fashion, and his music was not appreciated in a musical scene which was increasingly modernist even while Milford's own music was becoming more conservative.
As well as large scale works, Milford also wrote smaller pieces, for example organ pieces suitable for playing as church voluntaries (he was himself a village church organist) and piano works. Milford was able to show the character of a song setting with just a few notes, for example in the very brief piano introduction to If it's ever Spring Again.
Recordings of his music are few, although some of his music - some songs, his Mass for five voices Op. 84, Concertino Op. 106 and a selection of pieces including Fishing by Moonlight Op. 96 - are available.
Milford's Suite for Oboe and String Orchestra, a composition of extraordinary beauty, was recorded by Cameo Classics with British oboist John McDonough and the Malta Philharmonic Orchestra conducted by Michael Laus in 2011. The CD of premiere recordings (CC9032CD) is available from distributors, Nimbus Records.
A more complete list may be found in Copley (1984).
Milford's compositions include
- The Shoemaker Op. 3, children's opera (1923)
- Double Fugue Op. 10, for orchestra (1926)
- A Chorale Prelude on "St. Columba" Op. 14 (1928) for organ (arranged for orchestra as film music for a television episode of Star Trek)
- The Darkling Thrush Op. 17, for violin and orchestra (1929)
- Go Little Book Op. 18, suite for flute, optional soprano and orchestra (1928)
- Two Orchestral Interludes Op. 19e, for orchestra (arrangements of two easy piano duets, written before 1930)
- Concertino for Harpsichord and String Orchestra Op. 20 (1929)
- A Prophet in the Land Op. 21, dramatic oratorio (1929)
- Symphony Op. 34 (1933, perhaps never performed in full, withdrawn in 1956 although admired by Vaughan Williams)
- Miniature Concerto in G Op. 35, for string quartet or orchestra, with optional double basses (1933)
- Four Songs Op. 36 (1933) includes
- So Sweet Love Seemed (no. 1)
- Concerto Grosso Op. 46 (1936)
- Violin Concerto Op. 47 (1937)
- Four Hardy Songs Op. 48 (1938) includes
- The Colour, no. 2
- If it's ever Spring Again, no. 3
- Elegy for James Scott, Duke of Monmouth and Buccleugh Op. 50, for string orchestra (1939)
- A Mass for Children's Voices Op. 62 (1941–42)
- Sonata in C for flute and piano, Op. 69a (1944), of which Milford arranged the slow movement for flute and string orchestra
- Elegiac Meditation Op. 83, for viola and string orchestra (1946–47)
- A Mass for Christmas Morning Op. 84, for five voices (1945–47)
- Fishing by Moonlight Op. 96 for piano and string orchestra (1952 arrangement of 1949 piece for two harpsichords or two pianos)
- Festival Suite Op. 97, for string orchestra (1950)
- Overture for a Celebration Op. 103 (1952–54)
- Concertino in E Op. 106, for piano and string orchestra (1955)
- Three Airs, for treble recorder or flute and piano (1958)
- The Scarlet Letter Op. 112, opera based on novel by Nathaniel Hawthorne (1958–59)
- Copley, Ian (1984). Robin Milford. Thames Publishing, London. ISBN 0-905210-28-X.
- Blom, Erik (1942). Music in England. Penguin Books, London.