Constructed on the site of the Governors’ residence during the Spanish occupation of Gibraltar,  the library was officially opened in 1804 by the Duke of Kent. In 1823 the library's fees were "100 hard dollars" which was paid by the 150 proprietors of the "Commercial Library". Each proprietor was entitled to borrow one large or three smaller books or an entire set of a novel for one to two weeks. In exchange they had to also pay 16 dollars per year. This was a commercial affair and membership of the library could be bought or sold. The thirteen members of the committee were elected annually and the library was to be open seven days a week with both winter and summer hours.
The library served as the headquarters and archive service of the Gibraltar Chronicle, the world's second oldest English languagenewspaper. The Library was established for and by the officers of the Garrison of Gibraltar. It has remained a private entity run by a Trust for over two hundred years up until, September 2011, at which point the Library was transferred to the Government of Gibraltar.
The Garrison is a library of 45,000 books, including many rare volumes. This library exists to hold the collection which includes good coverage of the subjects of culture and travel. The library was started to occupy officers stationed in Gibraltar. It has an excellent local history collection. Many lithographs and art prints are held here and many of the furnishings have interesting historical backgrounds.