It was built in 12th century A.D (approximately 1143). When a Kakatiya king went hunting and was resting under a tree, he heard the chanting of Lord Rama's name. Wondering who it was in the middle of a dense forest, he discovered a stone idol of Lord Hanuman, in seated posture and the voice coming from the vigraha. Having paid his respects, he returned to his capital, and that night, Lord appeared in his dream and asked him to construct a temple.
The temple was built immediately and successive kings ruled it well. About 400 years later, Aurangzeb sent out his armies to all corners of the country to wipe out Hindus and Hindu temples. At this temple, the army couldn't even step close to the compound wall. When the general reported this to Aurangzeb, he himself went with a crow bar to break down the temple. At the threshold of the temple, he heard a deafening roar rumblning like thunder, and the crowbar slipped from his hands as he was shaking in fear. Then he heard a voice in the heavens "Mandir todna hai rajan, to kar man ghat" ("if u want to break down the temple king, then make ur heart stronger") which is why the place got the name kar-man-ghat. And to this day, Lord Anjaneya sits peacefully meditating and blessing devotees, as Dhyana Anjaneya Swamy.