Lytell (born James Sarrapede) had his first professional work at age twelve, and by the beginning of the 1920s he was recording in jazz ensembles. He played in the Original Indiana Five in 1921 and the Original Memphis Five in 1922–1925, and also played in the Original Dixieland Jazz Band in 1922–1924. After the 1920s, he rarely performed in jazz settings, spending more time as a studio and orchestra musician. He worked as a staff musician for NBC during this time, and in Johnny Green's orchestra in 1934–1935. From 1949 into the late 1950s, he appeared in the New Original Memphis Five revival band, and recorded with Connee Boswell in 1956. He continued to perform until a year before his death. As a leader, he recorded eighteen titles in 1926–1928 and six more for London Records in 1950.