It was Wales' only long-term, mature students residential education college and was established in 1927 by Thomas Jones (T. J.), Cabinet Secretary to both David Lloyd George and Stanley Baldwin, to continue the work of Workers' Educational Association in a residential environment, with Ben Bowen Thomas as its first warden. Plas Wernfawr was acquired at a knock-down price from a seller sympathetic to the project to be the base for the college.
Starting with just six students, mostly from the South Wales Coalfield area, numbers increased to 30 in the 1930s, 70 in the 1960s, serving the whole of Wales. Then, with Ieuan Jeffries-Jones as warden, Coleg Harlech began offering a two-year diploma course validated by the University of Wales, which became a preparation for university education for those who had missed out on earlier education: it became well known as a "second chance" college, often for people who, for economic or social reasons, never had a first chance.
By the 1980s and 1990s, higher education institutions generally were growing, and expanding access opportunities wider than before. This began to threaten Coleg Harlech’s niche, and ultimately Coleg Harlech, once funded as a unique institution in Wales, came under the funding regime with other further education colleges, and became less distinctive.
Coleg Harlech always had a close association with the WEA and merged with WEA (North Wales) in 2001 to become Coleg Harlech Workers' Educational Association (North Wales), subsequently merged into Adult Learning Wales - Addysg Oedolion Cymru, which now operates the site. The college's residential students were once supported financially by bursaries from the Welsh Government, previously the Welsh Office but as access to higher and further education widenened and the college's provision became less distinctive, these came to an end, in effect bringing about the termination of residential courses.
In February 2017 it was announced that Coleg Harlech would be closing as an adult education site at the end of the academic year. 
The institution has always been named, simply, Coleg Harlech: there is no ‘English version’ of the name.
The campus is centred on Plas Wernfawr, a house originally built in 1907–08 for George Davison, and designed in Arts and Crafts style by the radical architect George Walton. The building is in simple classical style, built of dressed blocks of local grey stone. The east front demonstrates strong horizontal lines, formed by two rows of sash windows, a projecting dentilated string course at eaves height forming the base of a pediment which contains a central oculus. Walton also laid out the garden. After it became Coleg Harlech, a library wing was added, designed by local architect Griffith Morris in an Art Deco style. Plas Wernfawr, together with the terraces revetment walls of the garden on the seaward side is a Grade II* listed building. The forecourt and garden structures on the inland side are listed Grade II.
Plas Wernfawr once held a collection of artworks which had been donated or purchased by the college over the years. However, a financial crisis at the college in 2013 forced the sale of these artefacts and many rare books from the college library.
The Great Hall, part of Walton’s original design, was destroyed by fire in 1968. It was replaced by a Brutalist theatre building, designed by Colwyn Foulkes & Partners. The theatre building was opened as Theatr Ardudwy in 1973 but has since changed its name to Theatr Harlech. It is operated by a separate body from the college.
The campus also includes a Gymnasium built c. 1970 to designs by Colwyn Foulkes and an Amenity Centre, designed in 1985 by the Percy Thomas Partnership. Other buildings including a concrete tower block once used for accommodation and Wern Fach, once the Warden's residence, are no longer in the ownership of the college.
- ”Miraculousy Unfolding Possibilities”:Coleg Harlech and Residential Adult Education, 1927–2007 in England (ed.), 2007
- 20th Century History of Harlech
- Haslam, Orbach and Voelker 2009 ‘’The Buildings of Wales: Gwynedd’’ Pevsner Architectural Guides, Yale University Press, p. 615
- Gwynedd Archaeological Trust, Historical Environment Record PRN 35122
- Gwynedd Archaeological Trust, Historical Environment Record PRN 35124
- The Public Catalogue Foundation (2013) Oil Paintings in Public Ownership in North and Mid-Wales, pp. 203–206
- BBC News. "Coleg Harlech art auction raises around £100,000 to clear debts". BBC News. BBC. Retrieved 5 January 2016.
- Gwynedd Archaeological Trust, Historical Environment Record PRN 35127
- Gwynedd Archaeological Trust, Historical Environment Record PRN 35128
- Davidson, Andrew (2008) Coleg Harlech: Archaeological Assessment Gwynedd Archaeological Trust report number 761
- England, Joe (ed.) (2007) Changing Lives: Workers’ Education in Wales 1907–2007 ISBN 978-0-9514580-1-3
- Stead, Peter (1977) Coleg Harlech: The First Fifty Years, ISBN 978-0-7083-0657-4
- White, Eirene (1977) Thomas Jones: Founder of Coleg Harlech, ISBN 0-901833-84-3