It was originally a "seigneurie" or Lord of the Manor and in turn this lordship was originally a dependency of the barony of Beauville. It was detached in favour of the De Ver family in 1463 whose descendants by marriage, the De Puybusque, became the Marquises of Toutens. The De Ver family had lived in a chateau in Toutens since before the mid 16th century, certainly before 1554 and possibly 1503 or even earlier. Then the marquis de Toutens were more plainly styled the Seigneur de Toutens, feudal Lords with considerable rights to collect income and levy justice in their fiefdom. Then on June 28, 1580, during the wars of religion, the Vicomte de Turenne a friend and ally of the future King Henry IV of France, led a Protestant (Huguenot) army and attacked several chateaus and forts in the area around Toutens. The chateau at Toutens was burned and razed. The present chateau was built likely in the early to mid 18th century by the De Ver family using some of the stone and other material from their feudal château but on another site some 300 meters to the west of the former site.