Born in Tipton County, Tennessee, Sharp's mother died when he was eight and his father moved the family to Texas. By the age of twenty Sharp was operating a water well drilling company with his brother. Sharp began drilling oil wells in 1893, although his first attempt was unsuccessful due to quicksand. Later innovations by Sharp would allow drilling through these conditions in 1901. Sharp also developed the Sharp-Hughes Rock Bit to drill through hard rock.
Sharp made his fortune from the trading of leases and contracting for oil wells. He was a founder of the Moonshine Oil Company and president of Producers Oil Company, and also held an interest in the Texas Company (later Texaco). He was also co-founder of the Sharp-Hughes Tool Company, which was renamed the Hughes Tool Company on Sharp's death in 1912 after Sharp's widow Estelle sold her shares to Hughes.