Born in Quierschied, Saarland, after playing football in his youth, Hary switched to sprinting at age 16. Only a few years later, in 1958, he won his first international title when he came first in the 100 m and the 4 x 100 m at the European Championships. He was also one of the first track stars to be affected by the rivalry between Adidas and Puma; each of the two then-fledgling companies wanted the "world's fastest man" to wear its shoes. Rumors of cash payments were floated, but no evidence was ever found to support the claim. Today, money to "support" athletes and Olympic teams are commonplace. This was not the case in Hary's time.
Also in 1958, Hary appeared to have run a new world record with a time of 10.0 seconds, but the track's slope of 11 centimetres (4.33 in) was found to exceed the maximum allowed 10 centimetres (3.94 in). In 1960, another try at the world record in Zurich, Switzerland seemed to have failed again, after judges believed Hary's start of another 10.0 race had been false—Hary was well known for his fast starts. In a re-run that same evening, Hary finally set the world record, which stood for just 24 days, but this performance stood as a European record for eight years less one day. His coach was Ante Tešija.
That same year, at the Olympics, he achieved his greatest moment of fame. After a nerve-wracking number of near-starts, Hary sprinted to the gold in 10.2 seconds to become the first non-American winner of the event since Canadian Percy Williams.
In the final of the 4 x 100 m, Hary and his teammates appeared to have finished second behind the team for the United States, but 15 minutes after the final it was announced that the USA had been disqualified for a faulty exchange. Germany's time, 39.5 seconds, equaled their own world record.
After the Olympics, Armin Hary was suspended by the German federation. He had demanded a payment to compete in a track meet (this in the days when the lines between professional and amateur were blurring) and when the promoter refused to pay, Hary refused to run. The German athletic federation suspended him, and Hary retired from the sport.