Kellmünz gets attention among the many settlements in the Iller valley due to its uncommon name, which points to a pre-Germanic past. The name derivates from "Caelius Mons" (engl. Heaven's Mountain). In 15 B.C. the Romans occupied the land between the Alps and the Danube and the province of Raetia was established. The legionaries renamed an initially Celtic settlement to Caelius Mons, probably referring to the Caelian Hill in Rome. After the fall of the Limes by the Alemanni the Romans changed their strategy to protect the Alpine foothills. Instead of a rampage at the border they built forts in the back-up area, so called "castellums". One of the most famous stood around 300 AD on the Caelius Mons. It was surrounded by a massive stonewall in the fourfold, 100 m long and 100 m wide with many towers. The castellum could accommodate 300 men. At the place, where the church stands today, was an equally large hall with an apse and a portico. That building was oriented to the south, while the church points to the east. The hall served mainly the administration, the jurisdiction and the cult.
Near the end of the war, on 25 April 1945, the bridge over the Iller was destroyed by the Wehrmacht, in order to prevent the allied troops the passing to Bavaria. After that combat actions took place, in which tanks of the U.S. Army shot into the village. Thirty-three estates were destroyed.