According to a story, he was initially a sarangi player, but decided to switch to vocals because of low status of sarangi players. In early years he used to sing with his brother Abdul Haq. They appeared before Murtuza Khan of the Maula Bakhsh khandan in order to appear before the Mahajara and were influenced by Murtuza's style.Baroda ruler was impressed by this singing duo and made them the court musicians. This is where he met Tarabai Mane, who was the daughter of Sardar Maruti Rao Mane, a member of the royal family. When they decided to get married, they were ousted from Baroda. The couple settled down in Bombay. In 1922 Tarabai Mane left Abdul Karim Khan, which apparently had a major impact on his music - making it pensive and meditative. Karim Khan's first wife, Gafooran was the sister of another Kirana master Abdul Wahid Khan, who was also his cousin.
Ustad Abdul Karim Khan was invited to the Mysore court where he met famous Carnatic music Carnatic masters which also influenced his music. In particular the singing of his Sargam was a direct influence of Carnatic practice. He became a frequent visitor to Mysore Darbar which conferred on him the title Sangeet Ratna. On the way to Mysore he used to stay with his brother in Dharwad where he taught his most famous disciple Sawai Gandharva. In 1900 for eight months he taught Kesarbai Kerkar, who would go on to be one of the 20th century's most renowned vocalists. In 1913 he founded the Arya Sangeet Vidyalaya in Poona to teach students. He would wholeheartedly teach all his students unlike other family ustads of the era. He finally settled down in Miraj till his death in 1937 when returning from a concert tour of the south. Every year in August commemorative music concerts are held in Miraj. The innovations he brought to his vocal style distinguishes Kirana style from others. The slow melodic development of the raga in Vilambit laya slow tempo was the most characteristic aspect of his music. He worked hard to maintain his voice to be sweet and melodious which shaped his music. The thumri style he developed as also quite different from the Poorab ang or Punjabi ang. His thumri progresses in a leisurely languor with ample abandonment. He was also the first Hindustani musician to seriously study the Carnatic system and probably the first to be invited to sing all over the south. He has even recorded a Thyagaraja Krithi. He was also influenced by Rehmet Khan of the Gwalior gharana and adopted the direct style of presentation.