Scarborough was born in St. Martin's-in-the-Fields, Westminster, in 1615, to Edmund Scarburgh and his wife Hannah (Colonel Edmund Scarburgh, prominent Virginia colonist, was his brother), and was educated at St Paul's School, Gonville and Caius College, Cambridge (BA, 1637, MA, 1640) and Merton College, Oxford (MD, 1646). While at Oxford he was a student of William Harvey, and the two would become close friends. Scarborough was also tutor to Christopher Wren, who was for a time his assistant.
Following the Restoration in 1660, Scarborough was appointed physician to Charles II, who knighted him in 1669; Scarborough attended the king on his deathbed, and was later physician to James II and William and Mary. During the reign of James II, Scarborough served (from 1685 to 1687) as Member of Parliament for Camelford in Cornwall.
Scarborough was an original fellow of the Royal Society and a fellow of the Royal College of Physicians, author of a treatise on anatomy, Syllabus Musculorum, which was used for many years as a textbook, and a translator and commentator of the first six books of Euclid's Elements (published in 1705). He also was the subject of a poem by Abraham Cowley, An Ode to Dr Scarborough.
Scarborough died in London in 1694. He was buried at Cranford, Middlesex, where there is a monument to him in the parish church erected by his widow.