Its history traces back to the 1200s. After centuries of being an exclusively boys school, in 1967 the then Ludlow Grammar School merged with Ludlow Girls High School, the whole retaining the name Ludlow Grammar School. A peculiarity of the boys Grammar School was that there was no 1st year, with new boys entering in 2nd year. Ludlow Grammar School was founded circa 1200, making it one of the oldest educational institutions in England. In 1977 the (merged) grammar school became Ludlow College, with secondary education in the town instead being provided only by the Ludlow Church of England School.
The original boys school is in ancient stone and brick buildings at the bottom of Mill Street, including the Palmers Hall, a Grade II* listed building. The girls school was based around an extensive Georgian red-brick mansion at the top of Mill Street/on Castle Square. Today three sites are used by the college - Castle Square (the former girls school), Mill Street (the former boys school) and Lower Mill Street.