After 1560, when Ieyasu abandoned the Imagawa, Kazumasa then became a valued retainer and administrator under him. During the year 1562, when Ieyasu managed to convince Imagawa Ujizane to release his family, Kazumasa acted as guardian of the Imagawa, which at the time was a very dangerous task.
After Toyotomi Hideyoshi's victory over Shibata Katsuie, in 1583, Ieyasu expressed his congratulations to Hideyoshi though Kazumasa. A later, Kazumasa and Sakakibara Yasumasa accordingly issued statements attacking Hideyoshi, due to the Tokugawa's decision. Kazumasa served at the Komaki headquarters during the Komaki-Nagakute Campaign.
In 1585, Kazumasa, very dismayed by what he saw as Tokugawa's foolhardy path of resistance against Toyotomi Hideyoshi, switched sides to Hideyoshi. This inconvenienced Ieyasu, who had to reconstruct his military organization and defensive policies, since Kazumasa had significant knowledge about their organization.
Kazumasa then later decided to retire and live with his son Ishikawa Yasumichi, until he died in 1609.