In 1885 he received his doctorate at Würzburg, and later worked with his brother, Salomon Herxheimer (1841-1899) in Frankfurt-am-Main. In 1894 he became director of the dermatology clinic in Frankfurt, and with Paul Ehrlich (1854-1915), was instrumental in founding the University of Frankfurt. In 1914 he became professor for skin and venereal diseases at the new university. In late 1942, at the age of 81, Karl Herxheimer was taken to the Theresienstadt concentration camp where he died.
Herxheimer is credited with providing an early description of acrodermatitis chronica atrophicans (diffuse idiopathic cutaneous atrophy), which is a dermatological disorder associated the latter stages of Lyme disease. This condition is sometimes referred to as Pick-Herxheimer disease, named along with co-discoverer Philipp Josef Pick (1834-1910). With Austrian dermatologist Adolf Jarisch (1850-1902), the Jarisch-Herxheimer reaction is named, which is an inflammatory reaction to Salvarsan, antibiotics or mercury, when using these agents to treat syphilis.