Elmes was educated at Merchant Taylors' School, and, after studying building under his father, and architecture under George Gibson, became a student at the Royal Academy, where he gained the silver medal in 1804. He designed a large number of buildings in London, and was surveyor and civil engineer to the Port of London, but is best known as a writer on the arts. He exhibited at the Royal Academy between 1801 and 1842.  He was vice-president of the London Architectural Society from its foundation in 1806.
In 1813-4 he restored the top part of the spire of Chichester Cathedral, reconstructing the pendulum device incorporated into it by Sir Christopher Wren to counteract the effects of strong winds. Elmes described the contraption in his biography of Wren, calling it "one of the most ingenious and appropriate of its great inventor's applications."
He was the founder and editor of the Annals of the Fine Arts, a quarterly magazine published between 1816 and 1820. The content of the periodocal was greatly influenced by the views of the historical painter Benjamin Robert Haydon, with whom Elmes had become friends while both were still students at the Royal Academy. Elmes claimed to have written the first review of Haydon's work ever published, in the Monthly Magazine in 1806. He also edited the Magazine of the Fine Arts and Monthly Review from 1821.
Elmes resigned from his post with the Port of London in 1848, due to a loss of sight, from which he later partially recovered.
His buildings included:
- St John the Evangelist's Church, Chichester (1812–13) built in the Classical-style as an Anglican proprietary chapel. The Grade I-listed building is now redundant and in the care of the Churches Conservation Trust.
- The New House of Correction, Bedford (1819). Sited on the main road from Bedford to Kettering, it was arranged on the panopticon system. The building was demolished 1851.
- A block of large terraced houses at 6–12, Queen Anne's Gate, Westminster (1837). Number 6 was built as the Parliamentary Agency Offices. Designed in collaboration with Harvey Lonsdale Elmes.
- A Letter to Thomas Hope, Esq., on the Insufficiency of the existing Establishments for Promoting the Fine Arts towards that of Architecture and it's Professors (1813).
- Hints for the Improvement of Prisons, and for a more Economical Management of Prisoners, partly founded on the Principles of John Howard (1817).
- New Churches. A Letter to the Earl of Liverpool on that part of the Speech of H.R.H. the Prince Regent which recommends the attention of Parliament to the Deficiency in the Number of Places of Public Worship belonging to the Established Church (1818).
- Lectures on Architecture, comprising the History of the Art (1821).
- Memoirs of the Life and Works of Sir Christopher Wren (1823).
- The Arts and Artists; or. Anecdotes and Relics of the Schools of Painting, Sculpture, and Architecture (3 volumes,1825).
- A General and Bibliographical Dictionary of the Fine Arts (1826).
- Metropolitan Improvements, or London in the Nineteenth Century (1827). Elmes' text accompanied a series of engravings after drawings by Thomas Hosmer Shepherd. John Summerson described the book as "a useful and nearly complete, guide to the new buildings of George IV's London", while regretting that the it was written "in the style of a facetious cicerone".
- A Practical Treatise on Architectural Jurisprudence (1827).
- A Practical Treatise on Ecclesiastical and Civil Dilapidations, Reinstatements, Waste, &c
- London Bridge; from its Original Formation of Wood to the Present Times with a Particular Account of the new London Bridge (1831).
- A Topographical Dictionary of London and its Environs (1831).
- A Guide to the Port of London (1832).
- Horas Taciva. A Thought-Book of the Wise Spirits of all Ages and all Countries, fit for all Men and all Hours (1852).
- Sir Christopher Wren and his Times (1852).
- Thomas Clarkson, a Monograph (1854).
- Encyclopaedia Britannica, 1911.
- Graves, Algernon (1905). The Royal Academy: A Complete Dictionary of Contributors from its Foundations in 1769 to 1904 3. London: Henry Graves. pp. 45– 6.
- "Literary and Philosophical Intelligence". The Monthly Magazine and British Register 21: 342. 1806.
- Corlette, Hubert C. (1901). The Cathedral Church of Chichester. Bell's Cathedrals. George Bell and Sons.
- Olney, Clarke (1952). Benjamin Robert Haydon, Historical Painter. University of Georgia Press.
- "James Elmes, Esq.". The Gentleman's Magazine: 784– 5. 1862.
- Elleray, D. Robert (2004). Sussex Places of Worship. Worthing: Optimus Books. p. 44. ISBN 0-9533132-7-1.
- "The Church of England Statistics & Information: Lists (by diocese) of closed church buildings as at October 2012" (PDF). Church of England. 1 October 2012. Archived from the original on 30 January 2013. Retrieved 30 January 2013.
- English Heritage. "Details from listed building database (1026696)". National Heritage List for England. Retrieved 31 March 2012.
- "Bedford's Gaol buildings". Bedfordshire Borough Council. Retrieved 10 August 2013.
- Hankin, Ralph B. (1828). An Account of the Public Charities of the Town of Bedford. Bedford. p. 94.
- English Heritage. "Details from listed building database (425004)". Images of England.
- List from: "James Elmes, Esq.". The Gentleman's Magazine: 784– 5. 1862.
- Summerson, John (1962). Georgian London. Harmondsworth: Penguin Books. p. 198.
- Baker, Anne Pimlott. "Elmes, James (1782–1862)". Oxford Dictionary of National Biography (online ed.). Oxford University Press. doi:10.1093/ref:odnb/8733. (Subscription or UK public library membership required.)
- Works by or about James Elmes in libraries (WorldCat catalog)
- The Arts and Artists, or Anecdotes and relics of the schools of painting, Vol. I, James Elmes, 1825, (John Knight & Henry Lacey, London)
- Sir Christopher Wren and his Times, James Elmes, 1852, (Chapman & Hall, London)