Carlebach belonged to a well known German rabbi family. His father Salomon Carlebach (1845–1919) was rabbi in Lübeck. He had seven brothers and four sisters. He attended the Katharineum school in Lübeck, where he befriended his schoolmate Thomas Mann, as the latter recalled.
Four of his brothers were rabbis as well. They are Emanuel Carlebach (1874-1927), Joseph Carlebach (1883–1942), David Carlebach (1885–1913) and Hartwig Naftali Carlebach (1889–1967). Carlebach is most known for his work in founding Orthodox Jewish schools in Germany, notably Leipzig, from 1900. He was a leading figure in the construction of The Jewish High School (Höhere Israelitische Schule) and the synagogueEtz Chaim.