Lee also narrowily missed out of making the 1948 Summer Olympics, losing to Jay Lambert in the finals of the US Olympic Trials. In 1950 he won the New York Golden Gloves Tournament of Champions in the heavyweight division, by KO over Nick Vasquez, the heavyweight Intercity Golden Gloves Championship, besting Kirby Seals and the National AAU Heavyeight Championship, when he decisioned Stan Howlett of Madison, IL in the final. He would repeat as National AAU Heavyweight champion again in 1951 and earn a bronze medal at the 1951 Pan American Games. Lee went to the 1952 Summer Olympics in Helsinki as a reserve Heavyweight. He was told that if he could make the weight limit, he could compete as a Light Heavyweight. Lee lost twelve pounds, and went on to defeat Argentina's Antonio Pacenza for the Gold Medal on a 3-0 decision. He was awarded the Val Barker Trophy for Outstanding Boxer at the 1952 Summer Olympics.
At the time of the Olympic Games, Lee had already earned a Master's Degree from Howard University. In 1948, he had been arrested in his hometown of Covington, when he became one of first blacks to sit in the all-white section of a bus.