Compton was born in Harrodsburg, Kentucky, and is first recorded as a pianist and entertainer in Louisville in about 1904. That year, he met pianist Tony Jackson, collaborating with him in writing a piece, "The Clock of Time", which was reportedly re-used as the basis of the 1922 song "My Daddy Rocks Me (With One Steady Roll)", recorded by Trixie Smith with a writing credit to music publisher J. Berni Barbour. Compton then worked as an itinerant piano player, working in Chicago in 1906 and also spending time in Wyoming, Washington, and New York.
He married singer Nettie Lewis in 1911, and started living with his wife and mother in Chicago, where he worked as a duo with Jackson as well as accompanist to his wife. He worked with composer Shelton Brooks, who dedicated his 1916 piece "Walkin' the Dog" to Compton. He also performed in San Francisco, where he met and worked with Jelly Roll Morton, Florence Mills, and Ada "Bricktop" Smith. Between 1917 and 1919 Compton and his wife also worked in Los Angeles and Seattle, before returning to Chicago.
Over the next few years he worked mainly in Chicago and Seattle, with small bands that, in Chicago, included singer Alberta Hunter, clarinetist Jimmie Noone and drummer Ollie Powers. In 1923, Compton recorded with Powers' Harmony Syncopators, later known as J. Glover Compton and the Syncopators, and also travelled to New York to record with Alberta Hunter. However, Compton failed to achieve the status of some of his peers, partly because he wrote little of his own material.
In 1926 he was invited by "Bricktop" to work for part of the year at her cafe, Chez Bricktop, in Paris, France. There, he appeared with the resident band, The Palm Beach Six, sometimes working with Sidney Bechet. He was shot in the leg by Bechet in 1928, apparently by accident when he tried to intervene in a fight between Bechet and another musician, and remained working in Paris after Bechet was expelled from the country. He returned to work in New York during this period, making his final trip to perform in Paris in 1939.
In the 1940s he returned with his wife to Chicago. He worked there again with Noone, and in 1949 he was interviewed by Rudi Blesh for his book They All Played Ragtime. In the early 1950s Compton opened his own bar in Chicago, and he was recorded performing there in 1956 for the album Meet Glover Compton. He suffered a debilitating stroke in 1957, and died in 1964, aged 80.