He was born on 2 April 1824 at Tilgate in Sussex into a family of wealthy Yorkshire industrialists, the son of James Armitage (1793-1872) and Anne Elizabeth Armitage née Rhodes (1788-1833), of Farnley Hall, just south of Leeds, Yorkshire. His great-grandfather James (1730-1803) bought Farnley Hall from Sir Thomas Danby (of the family of Thomas Dany, first mayor of Leeds) in 1799, and in 1844 four Armitage brothers (including his father) founded the Farnley Ironworks, utilising the coal, iron and fireclay on their estate. His brother Edward Armitage was a member of the Royal Academy.
He was raised at Avranches in France, and at Frankfurt and Offenbach in Germany. He attended the Sorbonne and King's College London. He became a physician, practising at the Marylebone Dispensary, in the Crimean War, and as a private consultant in London. He was forced to abandon his medical career because of deteriorating vision, eventually becoming blind.
Armitage decided to help make literature available to blind people through embossed type: in Britain this had become complicated by the proliferation of different standards. He formed the "British and Foreign Society for Improving the Embossed Literature of the Blind", later the "British and Foreign Blind Association for Promoting the Education and Employment of the Blind" and (after his death) the "National Institute for the Blind". This group decided to adopt the system of Louis Braille, and Armitage worked tirelessly for the adoption of Braille.
He was married to Harriett Black, and he died on 23 October 1890 at Thurles, County Tipperary, Ireland, following a riding accident.