Born in Charleston, South Carolina, but raised in Cleveland, Ohio, Smith's first instrument was clarinet and his education was in chemistry. He received his chemistry degree from Fisk University. Nevertheless in 1929 he became an alto saxophonist for Jimmie Lunceford's band, becoming one of the main stars in the group. In 1940 he led his own quintet as a side project. His success with Lunceford had lost its charms by 1942 as he now wanted more pay and less travel. He then switched to Harry James's orchestra, where he made more money, and stayed with him for seven years. After that he worked with Duke Ellington and Billy May. He was also part of the Gene Krupa Trio, and can be heard on the 1952 live Verve album The Drum Battle, part of the Jazz at the Philharmonic series (battle is with Buddy Rich). In 1954 he returned to Harry James's band. Added to all this he was involved in Jazz at the Philharmonic and worked with Nat King Cole.