Milman was the eldest son of Rev. Henry Hart Milman and Mary Anne Cockell, daughter of Lt Gen William Cockell. His father was rector of St. Margaret's, Westminster, and a Canon of Westminster Abbey, and later Dean of St Paul's. Milman was brought up at Ashburnham House and educated at Westminster School. He then went to Christ Church, Oxford where he pursued a successful rowing career. In 1845 Milman rowed for Oxford in the Boat Race and was a member of the Oxford University crew that won the Stewards' Challenge Cup at Henley Royal Regatta. He rowed in the Boat Race again in 1846 and at Henley won Stewards again and also won the Silver Wherries partnering Mark Haggard. In 1847 he won Stewards at Henley again, this time by row-over in a Christ Church four. In 1848 at Henley he was in the Oxford eight that won the Grand Challenge Cup, the Christ Church four that won Stewards and also won Silver Wherries again with Haggard. He was also President of the Oxford Union in Michaelmas 1849.
Milman took Holy Orders and in 1851 was mentioned in the letters of Thomas Babbington Macaulay with regard to his purchase of clerical vestments. Milman ordered a long vest of the latest clerical fashion which the tailor referred to as an "M.B waistcoat". When interrogated by Milman, the tailor confessed that "M.B." meant "The Mark of the Beast". On 1 October 1856, Milman was appointed librarian and chaplain of Sion College. He was rector of St Augustine and St Faith. In 1874 and 1875, he was president of Sion College. In 1880 he presented to the London and Middlesex Archaeological Society Some account of Sion College and of its library which was later published separately.
Milman remained librarian of Sion College library until his death in Kensington at the age of 83 in 1908. The library now forms part of the collections of Lambeth Palace Library and King's College London.
- Arthur Milman Henry Hart Milman:Dean of St Paul's; a biographical sketch 1900
- R C Lehmann The Complete Oarsman
- Henley Royal Regatta Results of Final Races 1839-1939
- Stephen Taylor From Cranmer to Davidson: a Church of England miscellany
- Sion College and library
- William Henry Milman Some account of Sion College ... and of its library: A paper read before the London and Middlesex Archaeological Society ... June 1880 The Society, 1890
- "Sion College and its library", Information Services and Systems, King's College, London.