The town dates from the late 17th century. It was originally set aside as the manor of the governor of the Province of New York and takes its name from the manor of Governor Thomas Dongan who named it for his home Castletown, in the town Kildrought (now Celbridge) in County Kildare, Ireland. The original manor house was constructed along Richmond Terrace, between Dongan Street and Bodin Street. It was destroyed by fire on December 25, 1878. The town was incorporated by the state of New York in 1788.