He was born in Neosho, Missouri to James Scott Sr. and Molly Thomas Scott, both former slaves. In 1901 his family moved to Carthage, Missouri, where he attended Lincoln High School. In 1902 he began working at the music store of Charles L. Dumars, first at menial labor, but before long demonstrating music at the piano, including his own pieces. Demand for his music convinced Dumars to print the first of Scott's published compositions, "A Summer Breeze", in 1903.
With the arrival of sound movies, his fortunes declined. He lost his theater work, his wife died without child, and his health deteriorated. Though it is said he continued to compose, he published nothing after Stark's retirement in 1922. He died in Kansas City, Kansas and was buried there in the Westlawn Cemetery.