During the Sengoku Period, Kakegawa was the location of a fortified settlement of the Imagawa clan. Following the defeat of the Imagawa at the hands of Oda Nobunaga at the Battle of Okehazama, Kakegawa, along with the rest of Tōtōmi Province came under the control of Takeda Shingen. Imagawa loyalist Asahina Yasutomo surrendered Kakegawa to Takeda ally Tokugawa Ieyasu. After the death of Takeda Shingen, Ieyasu took control of all of Tōtōmi Province, and assigned Kakegawa to his retainer Ishikawa Ienari. However, after the Battle of Odawara, Ieyasu was forced to surrender his domains in the Tōkai region to Toyotomi Hideyoshi in exchange for the provinces of the Kantō region. Toyotomi retainer Yamauchi Kazutoyo was assigned Kakegawa, and considerably improved on the structure of the castle and its moat system.
After the Battle of Sekigahara, Tokugawa Ieyasu regained control over the Tōkai region, and Yamauchi Kazutoyo surrendered Kakegawa in exchange for Tosa Province in Shikoku.
In February 1601, Hisamatsu Sadakatsu, Ieyasu’s half-brother, was created daimyō of the new Kakegawa Domain with revenues of 35,000 koku. In April 1607, he was reassigned to the newly-created Fushimi Domain, and turned Kakegawa over to his son Sadayuki, who ruled until reassigned to Kuwana Domain in 1617.
Ando Naotsugu was daimyō with revenues reduced to 28,000 koku from 1617 to 1619, when he was reassigned to Tanabe Domain as advisor to Tokugawa Yorinobu. Rule over Kakegawa was briefly given back to a branch of the Hisamatsu clan with the transfer of Hisamatsu Sadatsuna from Shimotsuma Domain in Shimosa Province with revenues slightly increased to 30,000 koku. However, he was transferred to Yodo Domain in Yamashiro Province in 1623.
The next inhabitant of Kakegawa Castle was Asakura Nobumasa, a close advisor of the ill-fated Tokugawa Tadanaga. After Tadanaga’s forced suicide, Asakura was stripped of his domain and exiled to Kōriyama in Yamato Province.
Aoyama Yoshinari, formerly of Hitachi Province was then assigned to Kakegawa, and his revenues were set at 33,000 koku. He was reassigned to Amagasaki Domain in 1635, and replaced by Matsudaira Tadashige, formerly daimyō of Tanaka Domain in Suruga Province. Domain revenues were increased to 40,000 koku, and after his death in 1639, he was succeeded by his son Tadamoto, who was then transferred to Iiyama Domain in Shinano Province only a month later. Honda Tadayoshi, grandson of Honda Tadakatsu was assigned to Kakegawa next, and revenues were increased to 70,000 koku. However, in 1644, Honda Tadayoshi was reassigned to Murakami Domain in Echigo Province.
Tanaka Domain again provided a successor, in the form of Matsudaira Tadaharu; however, the Bakufu reduced the rating of Kakegawa Domain back to 25,000 koku. In 1648, he was transferred to Kameyama Domain in Tamba Province. Tanaka Domain once more provided a successor: Hōjō Ujishige, who lasted until his death without heir in 1658.
In February the following year, Ii Naoyoshi was brought in from Nishio Domain in neighboring Mikawa Province, and governed Kakegawa until his death in 1672. His branch of the Ii clan continued to rule Kakegawa until 1706.
After a brief period under Matsudaira (Sakurai) Tadataka until his transfer to Amagasaki Domain, Kakegawa came under control of the generation of the Ogasawara clan from 1711 until their transfer to Tanakura Domain in 1746.
Hisamitsu clan (fudai) 1601-1624
Ando clan (fudai) 1617-1619
Hisamitsu clan (shimpan) 1616-1618
Asakura clan (fudai) 1624-1631
Aoyama clan (fudai) 1633-1635;
Matsudaira (Sakurai) clan (fudai) 1635-1639.svg){kind=small}
Honda clan (fudai) 1639-1644
Matsudaira (Fujii) clan (fudai) 1644-1648
Hōjō clan (tozama) 1648-1658
Ii clan (fudai) 1659-1706
Ogasawara clan (fudai) 1711-1746
Ōta clan (fudai) 1746-1868