Born in Uniontown, Alabama, Lipscomb never knew his father, and moved to Detroit with his mother at the age of 3. When he was 11, his mother was murdered in the neighborhood where they lived and he moved in with his maternal grandparents.
Lipscomb didn't attend college and was picked up as an undraftedfree agent by the Los Angeles Rams, who had first noticed Gene playing football in the Marines. He spent his service at Marine Corps Base Camp Pendleton, CA. He played for the Rams for three seasons from 1953-1955, before being traded to the Baltimore Colts for five seasons. In two of the seasons he played for the Colts, 1958 and 1959, he earned a spot in the Pro Bowl, and was instrumental in the Colts' two consecutive NFL Championships in 1958 and 1959. He then went on to play for the Pittsburgh Steelers for two seasons. During his '59-'60 & '60-'61 off-seasons, Lipscomb worked as a professional wrestler.