In 1929 Smith became an alto saxophonist for Jimmie Lunceford's band, becoming one of the main stars in the group. In 1940 he led his own quintet as a side project. His success with Lunceford had lost its charms by 1942 as he now wanted more pay and less travel. He then switched to Harry James's orchestra, where he made more money, and stayed with him for seven years. After that he worked with Duke Ellington and Billy May. He was also part of the Gene Krupa Trio, and can be heard on the 1952 live Verve album The Drum Battle, part of the Jazz at the Philharmonic series (battle is with Buddy Rich). In 1954 he returned to Harry James's band. Added to all this he was involved in Jazz at the Philharmonic and worked with Nat King Cole and Charlie Barnet.
Jazz critic John S. Wilson described Smith as "one of the triumvirate of great jazz alto saxophonists before Charlie Parker arrived. The other two were Johnny Hodges, who had a fat, luscious tone, and Benny Carter, a model of clean, pure-toned playing. Stylistically, Smith fell between Carter and Hodges for he combined some of Carter's clarity and singing directness with a variant of Hodges' gut sound."