He graduated from Yale University in 1849, where he assisted John Edmands, who was a student at the Brothers in Unity Library. Poole succeeded Edmands' position at the library and in 1848, while still a student, published his own 154-page index to periodical literature. A 524-page edition was published in 1853, and a third 1469-page edition in 1882.
He was assistant librarian of the Boston Athenaeum in 1851, and in 1852 became librarian of the Boston Mercantile Library. From 1856 to 1869 he was librarian of the Boston Athenaeum, where he inspired the careers of Charles Evans, William I. Fletcher, and Caroline Hewins. Poole was a pioneer in the public library movement. He was the first librarian of the Cincinnati Public Library from 1869 to 1873, where he successfully introduced the idea of opening the library on Sundays, and the first librarian of the Chicago Public Library from 1873 to 1887. Poole built the initial Chicago collection in part through persuading friends in the academic community across the United States to donate volumes. It did not hurt that his appeal suggested many books had perished in the great Chicago fire of 1871, even though the disaster had occurred two years before the city had begun a library. Poole capped his career as librarian of the Newberry Library, a private research institution, from 1887 to 1894. Poole designed the building, which still stands at 60 West Walton Street. While he was a moving force in the modern library movement, Poole's ideas ultimately put him on the wrong side of history. Poole believed each collection was unique and that librarians should design a building and catalogue system to fit his collection. The name of his contemporary, Melville Dewey, is attached to the idea of standardizing classification. Poole served as president of the American Library Association, and also as president of the American Historical Association.