Melchior came from a prominent Jewish family in Denmark. After a few years as rabbi in Germany, 1934 he became rabbi for the Jewish Community in Copenhagen From 1943 to 1945 he was acting rabbi for the Jewish refugees from Denmark in Sweden, and following the death of Max Friediger in 1947, became chief rabbi in Denmark. After World War II he pleaded for reconciliation with the new Germany.
Though his background was orthodox, his rhetorical skills and his culturally open attitudes made him a popular lecturer also outside Jewish circles.
He fathered four sons and two daughters, among them Arne Melchior (who became a frontbench Danish politician and government minister) and Binyamin (Bent) Melchior. In 1969, the latter succeeded his father as chief rabbi in Copenhagen, and is the father of Michael Melchior, who is chief rabbi in Norway and emigrated to Israel in 1986 where he became a member of the Knesset for the Labour-Meimad faction. He was also the grandfather of Rabbi Ove "Uri" Schwarz who served as the Chief Rabbi of Finland 1982–1987.