The Kuki were replaced by the tozamaNaitō clan, which ruled Toba to 1680. The domain then reverted to tenryō status under the direct control of the shogunate for one year. It then came under the control of the Doi clan (1681–1691), Ogyu-Matsudaira clan (1691–1710), Itakura clan (1710–1717), and Toda-Matsudaira clan (1717–1725) before finally coming under the Inagaki clan (1725–1871), where it remained until the Meiji Restoration.
As with all domains, Toba Domain was not a single contiguous holding, but was geographically scattered over a wide area. During the Bakumatsu era it ruled all of Shima Province (37 villages in Toshi District and 19 villages in Ago District) as well as 8 villages in Iino District, 4 villages in Taki District, and 5 villages in Watari District, all in Ise Province.
During the Boshin War, Inagaki Nagayuki remained loyal to the Shogunate, and as a result was fined heavily by the Meiji government and forced into retirement. His son, Inagaki Nagahiro became domain governor, and after the abolition of the han system in July 1871, Toba Domain became "Toba Prefecture", which merged with the short-lived "Watarai Prefecture" in November 1871, which later became part of Mie Prefecture.