Born in Austria to German parents, Grube returned to Germany in 1933. He attended the Schiller Gymnasium in Berlin. In 1955 Grube obtained a doctorate from the Freie Universität Berlin. He obtained a position in the Arts Library of the Staatliche Museen in Berlin, subsequently becoming assistant to Ernst Kühnel, curator of the Islamic collection and one of the first to rank Islamic art on an equal basis with classical and Western art. Under Kühnel, Grube's interest in Medieval art broadened to encompass Islamic art, which occupied him for the rest of his life during which he became recognized as one of the leading world authorities. In 1958 Grube obtained a one-year fellowship at the Metropolitan Museum of Art in New York, directed at that time by James J. Rorimer. He remained there until 1969, with an official appointment in 1962 in the Islamic department, of which he became the first curator in 1965 and where he was encouraged to build the collection. Grube held a simultaneous teaching appointment as Adjunct Professor of Islamic Art at Columbia University, In 1968 he was appointed Professor of Islamic and Far Eastern art at Hunter College in the City University of New York,
In 1959 Grube married Alberta Fabris in Verona, Italy. In 1972 he moved to Italy, first teaching in Padua and Naples, before being appointed as Professor of Islamic art at the University of Venice where he remained from 1977 to 1988. From 1972 to 1978 he served as a member of the Italian Archaeological Mission to Iran. On his retirement in 1988, he moved to London with his second wife, the art historian Eleanor Sims, an expert on Persian painting. In London he continued to publish extensively on Islamic art, acting also as editor of a journal he cofounded with Sims.