Nathaniel Sylvester (1610-1680) was the first European settler of Shelter Island. An Anglo-Dutch sugar merchant, in June 1651 with his brother Constant and two other men he purchased the whole of Shelter Island from the Manhanset Indians, whose sachem, or chief, was called "Yoki." The Shelter Island enterprise involved barrel-making, using the stands of white oak for shipping the West Indies tobacco, sugar, molasses and rum back to England. Sylvester used both local Native Americans and enslaved Africans from Barbados in his operation.
Between 6 July (date of marriage jointure) and 8 August 1653 (date of letter mentioning his changed condition because of marriage) he married Griswell Brinley, daughter of Thomas Brinley, one of the Auditors General of the Revenues for Charles I, and later for Charles II. Griswell was a younger sister of Anne Brinley, who in England had married William Coddington of Rhode Island in January 1650. When the Coddingtons returned to Rhode Island in mid-1651, Grizzell came along as a ward of Coddington. The Sylvesters were friends with Quaker founder George Fox, whom he entertained on at least one occasion on Shelter Island. They offered a place of refuge for several of the persecuted early Quakers in New England. Shelter Island families descended from Sylvester include Dering, Sprague, L'Hommedieu, Havens and Hudson.
A contemporary archaeological dig, the Sylvester Manor Project, a project overseen by the University of Massachusetts Boston, seeks to shed light on the Sylvester estate as it existed in the 17th and 18th centuries.