Svenson was born in Sweden, to Lola Iris Viola (née Johansson), a big band leader, actress, and singer, and Birger Ragnar Svenson, a personal driver, athlete, and bodyguard for the King of Sweden. He emigrated to the United States and then, when he was 17, joined the United States Marine Corps, serving until his honorable discharge six years later. His first state of residence in the United States was Georgia, where he became familiar with the rural Southern accent he later employed in some of his roles.
He also holds a fourth degree (Yondan = 4th Dan) black belt in judo. He earned his first degree (Shodan) belt at Kodokan in Japan, the home dojo of Judo, while stationed in Japan in 1961 as a Marine. He was the 1961 Far East Judo Champion in the Heavyweight Division.
In the late 1960s, Svenson had a recurring role in the hit TV series Here Come the Brides as Lumberjack Olaf "Big Swede" Gustavsen. In the mid-1970s, he took over the role of lawman Buford Pusser from Joe Don Baker in both sequels to the hit 1973 film Walking Tall, after Pusser himself, who had originally agreed to take over the role, died in an automobile crash. He reprised the role again for the short-lived 1981 television series of the same name.
One of his most famous roles in films was as murder witness-turned-vigilante Michael McBain in the 1976 cult classic Breaking Point. He played the Soviet agent Ivan in the Magnum, P.I. episode "Did You See the Sunrise?" (1982) and many years later had a cameo as an American colonel in Inglourious Basterds, as a tribute to his role in The Inglorious Bastards; he is the only actor to appear in both films.