He began his coaching career in 1964. In 1969 he was appointed as head coach of Maccabi Tel Aviv, with which he won 14 championships and the European title in 1977. As coach of the Israeli national team, he won second place in the EuroBasket in 1979 and sixth place in 1981 and 1983. In 1983, he unexpectedly announced his appointment as the coach for the West German national team and for BSC Saturn Koln, a position he held until 1985.
Klein led the West German national team to an eighth place finish in the 1984 Olympic basketball tournament in Los Angeles and a fifth place at the 1985 European championship held on West German home soil.
In 2007, he was diagnosed as suffering from Colorectal cancer and was believed to be on his deathbed. However, his health improved and he even went back to coaching.
In 2010 the filming of "Playoff" began, a movie based on Klein's life story, and directed by Israeli movie director, Eran Riklis.
Israeli basketball team, Elitzur Elkana, whom Klein coached towards the end of his life, is named after the veteran Israeli coach (Elizur "Ralph Klein" Elkanah). The basketball team also wrote a Sefer Torah in the name and honor of Klein.
The veteran group of Maccabi Tel Aviv basketball team commemorates the achievements and memory of Ralph Klein, and since his death it is named "Maccabi Ralph Klein Tel Aviv".