Rosbaud's first professional post was in Mainz, starting in 1921, as the music director of the city's new School of Music, which included conducting the municipal symphony concerts. He became the first chief conductor of the Hessicher Rundfunk Symphony Orchestra (later the Hr-Sinfonieorchester) of Frankfurt in 1928. During the 1920s and 1930s, he presented premieres of works by Arnold Schoenberg and Béla Bartók. During the Nazi era, his freedom to present new music was restricted. In 1937, he became the general music director of the city of Münster. In 1941, Rosbaud took the same position in Strasbourg, heading the Orchestre philharmonique.
In 1945 he was named music director of the Munich Philharmonic by United States occupation authorities. In 1948, Rosbaud's contract with the Munich orchestra was allowed to lapse because the city authorities wanted to move the orchestra's repertoire in a conservative direction. That year Rosbaud became the first chief conductor of the South West German Radio Orchestra in Baden-Baden, where he remained for the rest of his life. In 1954, he conducted the first performance of Schoenberg's opera Moses und Aron on 8 days' notice, and this performance was preserved on a 1957 commercial recording for Philips. He regularly took the SWR Symphony Orchestra to festivals of contemporary music, such as at Donaueschingen. He died in Lugano, Switzerland.
Gramophone recently remarked that Rosbaud "was one of the unsung heroes of mid-20th-century music, who ... gave thoroughly rehearsed and assimilated performances and premieres of the widest possible range of music".  Prominent in his legacy are recordings of the music of Bruckner, Mahler, Stravinsky and Boulez. A tireless advocate of new music, he was closely associated with Karl Amadeus Hartmann, conducting premiere performances of Hartmann's opera Simplicius Simplicissimus and his Second and Fourth Symphonies, amongst others.