The château was the seat of the Duke de Sully, Henri IV's minister Maximilien de Béthune (1560–1641), and the later dukes of Sully. It is a château-fort, a true castle, built to control one of the few sites where the Loire can be forded.
The site has perhaps been fortified since Gallo-Roman times, and was first mentioned in a document of 1102. In 1218, Philip Augustus constructed a cylindrical keep to the south of the present enclosure, of which buried foundations remain. Guy de la Trémoille, inheriting the fortress, undertook the construction of the donjon, flanked by four towers, beginning in 1395. In the sixteenth century the duke de Sully added the Petit Château to one side of the castle to provide more agreeable accommodation. Sully bought the domaine in 1602, enlarged the park and the fortress; he strengthened the embankments of the Loire to protect the town from occasional flooding. Henri IV never visited, but Mazarin and Anne of Austria took refuge here in March 1652 during the rigors of the Fronde, France's civil war. Turenne stayed here the same year, before his defeat of the Grand Condé at the battle of Bléneau. Later, in 1716 and again in 1719 the château sheltered Voltaire, when he had been exiled from Paris for affronting the Régent, Philippe, duc d'Orléans.
The Château de Sully-sur-Loire remained in the possession of the family until 1962 when it became a property of the Département du Loiret, and has since benefited from numerous restorations. It hosts a classical music festival each June. The château contains numerous tapestries (including a set of six seventeenth-century hangings, the Tenture de Psyché), paintings of Sully's ancestors and heirs, and seventeenth-century furnishings. Here is also the tomb of Sully and that of his second wife.