Joe Martin was born February 1, 1916, in Colorado. He was orphaned before he reached a year old and was raised by an aunt in Phoenix, Arizona. He married Christine Fentress on March 4, 1941, and had one child, Joe Jr., who himself became a national Golden Gloves champion.
He came to Louisville, Kentucky, in 1937 and joined the Louisville Police Department, serving until his retirement in 1974. In 1938 he became a boxing coach at the Columbia Gym in Louisville, where, in 1954, he began coaching three-time world heavyweight champion Muhammad Ali, who was then known as Cassius Clay. During this time, he also trained Jimmy Ellis as well as eleven National Golden Gloves champions.
In 1954, a twelve-year-old then known as Cassius Clay approached Martin to report that his bicycle had been stolen and told Martin that he wanted to "whup" the thief. Martin offered to teach him how to box and guided his career for the next six years. As a 1960 Olympic coach, Martin accompanied the champion to the Olympic Games in Rome, Italy, when Ali won a gold medal. In the 1950s Martin helped produce a weekly television show on WAVE-TV called Tomorrow's Champions, which was broadcast for twelve years. After winning the gold medal, Ali began his professional career but maintained contact with Martin until his death. In 1977, Martin was inducted into the Amateur Boxing Hall of Fame.