The town was established in 1843 in South Africa's inland plateau. It was founded to meet the religious needs of a growing farming community, but unlike most Karoo towns the church was not built as the physical focal point of the village; rather, the centrepiece is the village square. Unusually for this arid region, it was built astride a river which has been cited as the reason for the irregular street grid. The town has a substantial number of well-preserved houses and public buildings of Victorian and Edwardian Karoo style, with additions of verandahs dating from the 1920s. The Dutch Reformed Church building, the spiritual if not the physical centre around which the town evolved, dates from 1847, with a tower completed in 1909. It celebrated a centenary in 2009.
The naming of the town originated in the desire of the townsfolk to honour the new Governor of the Cape, Sir Peregrine Maitland, who took office in 1844. Maitland declined, however, suggesting instead that it be named after his father-in-law, the Duke of Richmond. It was officially named Richmond in October 1845. It was a resort town for European aristocratic patients of lung diseases such as whooping cough and tuberculosis in the 1800s due to its clean air and mineral rich waters.
Richmond was the birthplace of medical pioneer Dr Albert Hoffa, hailed as the founder of modern orthopaedics, author of a famous text book and some of whose techniques are still in use. Born in Richmond on 31 March 1859, the son of a local doctor, he studied medicine in Germany. In 1886 Hoffa was appointed as lecturer at the University of Würzburg, later becoming Professor and moving to Berlin. He died in 1907.