Dillard was born in Cleveland, Ohio, attended East Technical High School. He entered Baldwin-Wallace College in 1941 and joined Pi Lambda Phi International Fraternity, and two years later was drafted into the Army serving in the all-black 92nd Infantry Division known as the Buffalo Soldiers. He returned to college in 1946 and resumed athletics, to which he had been inspired by Jesse Owens, who was also from Cleveland and had attended East Technical High School as well. He won the NCAA and AAU 120-yard and 220-yard hurdles in both 1946 and 1947 and he tied world records in both events with a 22.3 in the 220 in 1946 and a 13.6 in the 120.
At the trials for the 1948 Summer Olympics, however, Dillard failed to qualify for the 110 m hurdles event, though he qualified as third (and last) for the 100 m, not his specialty.
At the Games, Dillard reached the final, which seemed to end in a dead heat between Dillard and another American, Barney Ewell. The finish photo showed Dillard had won, equalling the World record as well. This was the first use of a photo finish at an Olympic Games. As a member of the 4 x 100 m relay team, he won another gold medal at the London Games.
Four years later, still a strong hurdler, Dillard did qualify for the 110 m hurdles event, and won the event in Helsinki. Another 4 x 100 m relay victory yielded Dillard's fourth Olympic title.
Dillard attempted to qualify for a third Olympics in 1956, but failed.