Allen was born in Kanpur, then known as Cawnpore, in British India, where several generations of his family served under the British Raj. He was not born of an Anglo-Indian family, as stated above, but of British parents both born in India, whose ancestors had lived and worked in India for several generations. Following Independence in August 1947 the family moved back to England but when his father was asked to return to India to resume his job as a political officer Allen and his brother remained in England for their schooling, which in his case ended when he was 17 without academic qualifications. Allen spent three years as a tea taster in the City before setting about educating himself in order to become a teacher. A year in Nepal as a teacher with Voluntary Service Overseas (1966-67) convinced him that his future lay in history and travel writing, although it was not until 1974 that he met with any public success through his involvement with the BBC Radio 4 oral history series and book Plain Tales from the Raj. Since then Allen has spent long periods of his adult life in India as a traveler and writer of history.
His work focuses largely on India and the Indian Subcontinent in general. Allen's most notable recent work is Kipling Sahib: India and the making of Rudyard Kipling, a biography of Rudyard Kipling. His most recent work, Ashoka: the Search for India's Lost Emperor, was published in February 2012. He is currently working on a biography of Brian Hodgson, the 'father of Himalayan studies', to be published in September 2015 by Haus Books under the title The Prisoner of Kathmandu:Brian Hodgson in Nepal 1820-1843.