Prior to 1878 written evidence exists of a Llandovery rugby team, though in these reports it is not clear if this was a town or college team. On 22 February 1878 a report in the Carmarthen Journal refers to a rugby game between 'Llandovery' and other local teams but these were specifically named as college teams while Llandovery was not. In 1879 there is proof that Llandovery, separate from Llandovery College, played what may be seen as their first recognised official match against Christ College, Brecon. Llandovery became an inaugual member of the Welsh Rugby Union at a meeting held at the on 12 March 1881 in the Castle Hotel, Neath. In 1894, Llandovery was one of ten clubs who became affiliated to the Welsh Rugby Union, but gave up this status in 1910.
In November 1948 Llandovery RFC officially reformed, and began laying down the foundations that would give it future stability. Llandovery RFC moved their club headquarters to the White Hall and in 1956 the town's Improvement Committee purchased Barlow's Field which was levelled and reseeded. In the subsequent two years improvements included the building of changing rooms. In 1956 Llandovery RFC reapplied for membership to the WRU which was granted in 1957.