He was born in Tompkinsville, Kentucky, United States, moving to Indianapolis with his family as a child. He first played in a band with Erroll "Groundhog" Grandy who mentored J. J. Johnson and Wes Montgomery. From 1938 to 1940, Coe was with Buddy Bryant's band and by the age of 20, was already touring with the Jay McShann band, which included Charlie Parker, Al Hibbler, Walter Brown, Bernard Anderson, Gene Ramey and Harold "Doc" West.
In the 1950s, Coe recorded for King as a member of Tiny Bradshaw's band, then made a session with his own combo (though the company oddly insisted on billing him as Jimmy "Cole.") In 1953, States recorded his Gay Cats of Rhythm. In the late 1950s, Coe led the house band for the small Indianapolis-based label Note Records; some of the material was licensed to Checker, which had better distribution.
With his mid-1960s big band, Coe backed performers including Aretha Franklin, Roy Hamilton, and Gladys Knight & the Pips. Other musicians he worked with included Montgomery, Slide Hampton, David Baker, Freddie Hubbard, pianist Carl Perkins, Larry Ridley, Leroy Vinnegar, and doo-wop performers, The Students.
He died in Indianapolis in 2004, at the age of 82.