Zardis played bass from a young age, and studied under Billy Marrero of the Superior Orchestra. In his teens he was involved in a fistfight at a New Orleans theater, which resulted in his being sent to the Jones Waif Home. While there he began playing with another of the Home's residents, Louis Armstrong. He joined Buddy Petit's orchestra at age 16, and worked as a bassist in nightclubs and a tubist in brass bands in New Orleans in the 1920s, playing with Kid Rena, A.J. Piron, Punch Miller, Kid Howard, Jack Carey, Fate Marable, and Duke Dejan's Dixie Rhythm Band.
He was given the nickname "Little Bear" by Fats Pichon, a bandleader with whom Zardis played on the riverboat S.S. Capital in the 1930s. During that decade he also played with Count Basie in New York City, and recorded with George Lewis and Bunk Johnson. During the Second World War Zardis served in the Army, then worked briefly as a sheriff in the Western United States. Upon his return to New Orleans he played with Andy Anderson, but quit music between 1954 and 1964. Zardis worked as a Jailer in Jefferson Parish for several years.
When he returned to active performance, Zardis played often at Preservation Hall with Lewis and Percy Humphrey among many others. He continued to be a fixture of the New Orleans jazz scene up until his death in 1990, including several international tours.
Zardis was a master of the original New Orleans - style slap bass, achieving both clarity of intonation and a strong percussive beat. His skill placed him easily on a par with better known New Orleans slap bassists of his era like Pops Foster and Wellman Braud. Unlike Foster and Braud, however, Zardis remained in his home-town throughout his playing career and consequently was not as widely recorded or appreciated. Playing un-amplified string bass using gut strings in large halls or rooms with quirky acoustics posed significant challenges to New Orleans bassists. The short-statured Zardis, who was a powerful player and a creative soloist, overcame these handicaps by frequently turning his back to the audience in order to face his instrument into the back wall of the bandstand, thus bouncing his notes off the wall, and projecting his sound up and over the competing sounds of the louder front-line brass horn players and out over a room full of loud patrons. He employed this simple trick night after night during his years performing at Preservation Hall on St. Peter Street in the French Quarter.
Zardis was regularly featured in documentaries; he is himself the subject of three of them, Liberty Street Blues, Chester Zardis: Spirit of New Orleans, and Three Men of Jazz.