Rumpf was born in Berlin; during 1917 - 1920, he was educated as an insurance assessor, and worked until 1932 as an accountant, cashier and a foreign trade correspondent. In 1920 he joined the German Communist Youth Association, and in 1925 the Communist Party of Germany. From 1933 to 1938, he was arrested in penitentiary and detained at the Sachsenhausen concentration camp. After that he returned to work, and was a member of the resistance group Robert Uhrig. During 1944 - 1945, he was again in custody.
From 1945 to 1947 he was deputy leader of the finance department of the magistrate of greater Berlin, from 1947 to 1948 leader of the trust administration of Berlin, from 1948 to 1949 leader of the central administration for finances of the German Economic Commission.
In 1946. Rumpf became a member of the Socialist Unity Party of Germany (SED). During 1949 - 1967, he was a delegate of the Volkskammer, from 1950 as a candidate and from 1963 as a member of the central committee of the SED. He was from 1955 to 1966 Finance Minister and member of the cabinet council, and from 1963 its president.