Halvorsen was born in Sarpsborg, the son of baker Christian Halvorsen and Jakobine Dorthea Thronsen. He established himself as a ship broker in Hamburg, and his first marriage was with a German citizen. He married for a second time in 1951 with Sigrid Helga Willadsen.
Halvorsen was a centre-half who played 19 times for the Norwegian national team, and won the Norwegian Cup in 1917 as captain of Sarpsborg FK. Aged 18 years and 318 days, he is the youngest captain in a Norwegian Cup final. He made his debut for the national team in 1918, in a match against Sweden, and played four matches for the national team in 1918, and four matches in 1919. He represented Norway at the 1920 Summer Olympics in Antwerp, where the Norwegian team reached the quarter finals. Between 1922 and 1934, Halvorsen played in Germany for Hamburger SV, where he won the German championships in 1923 and 1928. He played his last match for the national team against Germany in Hamburg in 1923, his nineteenth game for the national selection.
He returned to his home country in 1934, and was hired as secretary of the Norwegian Football Association, NFF. This job also made him head of the national team's selection committee, and in the years before World War II, he also acted as national team coach.
With Halvorsen at the helm, Norway won the Bronze medals at the 1936 Olympics, and qualified for the 1938 World Cup. This was Norway's first and only appearance in the World Cup finals until the 1990s.
After the war, Halvorsen continued working for the NFF, with the title of Secretary General, until his death in 1955. He was a board member of Norsk Tipping from 1946. He was decorated Knight of the Swedish Order of Vasa. He died in Narvik in 1955.