Milford was born in Exeter, Devon, England, the eldest son of Samuel Frederick Milford, D.L., of Heavitree, Devonshire. He received his preliminary education at the High School, Exeter, and afterwards graduated M.A. at St John's College, Cambridge. He was called to the Bar at Lincoln's Inn, London, and practised his profession for several years at Bristol, where he held the appointment of judge of the Diocesan Ecclesiastical Court.
Owing to bad health, he was induced to seek an appointment in Australia. Through the influence of his cousin Sir William Webb Follett, the then Attorney-General of England, he was appointed Master in Equity of New South Wales, and left London in September 1842 for Sydney. He landed on 1 January 1843, and held the post of Master in Equity, together with that of Chief Commissioner of Insolvent Estates, until his appointment as Resident Judge in the district of Moreton Bay (now Queensland) in January 1856. Then he returned to Sydney in February 1859, and was a Supreme Court Judge till his death on 26 May 1865 in Maitland, New South Wales. He held also during this period the offices of Judge of the Court of Vice-Admiralty and Primary Judge in Equity.