His father, the Rev. James Nicol (1769–1819), was minister of Traquair, and acquired some celebrity as a poet. Educated at Edinburgh University (1825), James Nicol attended the lectures of Robert Jameson, gaining a keen interest in geology and mineralogy. He pursued these studies in the universities of Bonn and Berlin.
After returning home he worked zealously at the local geology and obtained prizes from the Highland Society for essays on the geology of Peeblesshire and Roxburghshire, areas of the Scottish Borders. He subsequently extended his researches over various parts of Scotland, and in 1844 published his able Guide to the Geology of Scotland.
In 1847 he was appointed assistant secretary to the Geological Society of London, in 1849 professor of geology in Queen's College, Cork, and in 1853 professor of natural history in the University of Aberdeen, a post which he retained until a few months before he died.
During these years he carried out important researches on the Southern Uplands of Scotland and on the structure of the Highlands. In the former region he gave the first clear account of the succession of the fossiliferous Lower Palaeozoic rocks (1848–1852). When he came to deal with the still older Highland rocks he made out the position of the Torridonian sandstone and Durness limestone, and their relations to the schists and gneisses. Nichol's matured views, although correctly recognising the fallacy in the extant theory of Roderick Murchison, were subsequently superseded by the correct theory of Charles Lapworth which was corroborated by Benjamin Peach and John Horne.
The more important of his papers were:
- "On the Structure of the North-Western Highlands" (Quart. Journ, Geol. Soc., 1861)
- "On the Geological Structure of the Southern Grampians" (ib., 1863)
He contributed the article "Mineralogy" to the ninth edition of the Encyclopædia Britannica. Among his other works were:
- Manual of Mineralogy (1849)
- Elements of Mineralogy (1858, 2nd ed., 1873)
- Geological Map of Scotland (1858)
- Geology and Scenery of the North of Scotland (1866).