Zimmer studied the Humanities and philosophy at Ellwangen, theology and jurisprudence at Dillingen and was ordained a priest on 1 April 1775. In 1777, he became repetitor of Canon law at the College of St. Jerome at Dillingen, and professor of dogmatic theology at the University of Dillingen in 1783. He was also appointed pastor of Steinheim in 1791. In 1795 he was dismissed from the faculty of the university, ostensibly because as pastor of Steinheim he should reside at that place, but in reality, because of his extreme idealisism. In 1799 he was appointed professor of dogmatic theology at the University of Ingolstadt, and when this university was removed to Landshut the following year, he was transferred there in the same capacity.
Though Zimmer rendered great service to the Catholic Church and religion by his fearless and successful combat against the Kantian Rationalism which was prevalent at Ingolstadt, he was himself a passionate adherent of the idealistic pantheism of Schelling, without, however, compromising his Catholic convictions in practice. To lessen the danger of inculcating his philosophical tenets in his lectures, he was relieved of the professorship of positive theology and given that of Biblical archaeology and exegesis in 1807. In 1819 he became rector of the university and deputy to the Second Chamber of the Bavarian Parliament. His chief theological work, "Theologiae christianae specialis et theoreticae" (4 parts, Landshut, 1802-6), is to a great extent permeated with Schellingian pantheism.