Born to a farming family in the county of Appomattox, Virginia he claimed to have learned to play the banjo from local African-Americans. He is the earliest known person to have played the banjo on stage. Aside from his important role in popularizing the instrument, he has often been credited with advancing the physical development of the modern five-string banjo. Whereas the instrument's resonating chamber had formerly been constructed from a gourd (like the banjo's African ancestors and cousins), Sweeney popularized the use of a drum-like resonating chamber (legend has it that he adapted a cheese box for this purpose). He has also been credited with adding the banjo's fifth string, which according to legend was for an instrument he created for his niece between 1831 and 1840. He supposedly added the fifth string because he was "allegedly unhappy with the limited rhythm and melodic variation of the four-string banjos popularly in use."  In fact, there is no proof that Sweeney introduced either innovation. "Outlandish claims have been made about Sweeney, from his being the "inventor" of the banjo to his being the first white man to play the banjo. These claims are part of an effort, beginning in the nineteenth century, to divorce the banjo from its African American origins."
Many people assume Sweeney's "fifth string" is the drone string—the chanterelle—but if Sweeney did really add a fifth string, it would have been another, lower string. That would provide more melodic and harmonic expression, as the quote suggests. The high-pitched, thumb or drone string (the chanterelle, the fifth on a modern banjo) is seen on surviving 18th-century four-string banjos, and in banjo illustrations that long pre-date Sweeney's heyday. The fifth string gives the banjo a reentrant tuning, a feature that existed in instruments of the ancient Greeks.
Sweeney began performing with the banjo in the early 1830s. He first performed throughout central Virginia for county court sessions. A few years later he joined a circus and traveled throughout Virginia and North Carolina. By 1839, Sweeney was performing in various blackface venues in New York. His earliest documented use of the banjo on stage was in April 1839. That same month, he performed alongside James Sanford at the Broadway Circus in New York with a blackface burlesque of The Dying Moor's Defence of His Flag called "Novel Duetts, Songs, &c". This was accompanied by a "Comic Morris Dance by the whole company". According to Billy Whitlock of the Virginia Minstrels, Sweeney gave Whitlock a few banjo lessons around this time.
In colonial America the banjo was known as an instrument of "the lower classes," but by 1841, Sweeney was remaking the banjo into an instrument for the middle class. His advertisements boasted that he played with "scientific touches of perfection". Another raved, "Only those who have heard Sweeny [sic] know what music there is in a banjo." For the next few years, he was the benchmark against whom other banjo players were compared. After a performance by Dan Emmett at the Bowery Amphitheatre Circus, the New York Herald wrote, "Emmit's [sic] banjo playing is fully equal to Jo [sic] Sweeney's, and far ahead of any other now in the United States." "Jenny Get Your Hoe Cake Done" and "Knock a Nigger Down" became two of Sweeney's signature tunes.
Sweeney saw success, and by early 1843, he embarked on a European tour that included stops in London and Edinburgh. In July 1843, Sweeney played during entr'actes at the Adelphi Theatre in Edinburgh, Scotland. Frank Brower of the Virginia Minstrels met him there and joined Sweeney's act as a bones player. The two toured, performing in early October at the Theatre Royal in Birmingham and later that month in a circus at Leicester. At some point, Brower parted company to tour with Dan Emmett, though he rejoined Sweeney by spring of 1844.
At this time, Dick Pelham met up with Sweeney and Brower, and the trio decided to reform the Virginia Minstrels with Sweeney as banjoist. They found Emmett in Bolton and talked him into joining, although Sweeney would be the troupe leader. The new Virginia Minstrels performed in Dublin at the Theatre Royal from 24 April to 7 May during entr'actes, then continued for a series of entr'actes and complete minstrel shows in Cork, Belfast, then Glasgow by the end of May. They did several shows at the Theatre Royal, Adelphi, and later in the Waterloo Rooms in Edinburgh, followed by a return engagement in Glasgow, this time at City Hall.
Joel Sweeney's younger brothers, Sampson ("Sam") and Richard ("Dick"), and his sister Missouri were also talented banjo and fiddle players. In 1845, Joel, Sam, and Dick formed the minstrel troupe "Old Joe's Minstrels". Joel died in Appomattox on 29 October 1860 of dropsy. Dick also died in 1860.
Sam Sweeney and his banjo playing came to the attention of the famed cavalry general J.E.B. Stuart. Stuart had Sweeney attached to his staff, which became one of Stuart's best-known personal quirks. Sam Sweeney's signature song, possibly penned by Stuart himself, was "Jine the Cavalry." Sam Sweeney died, possibly of smallpox, on January 13, 1864, and was buried in Graham Cemetery, Orange, Virginia.
- Cockrell 148.
- Rice 22..
- Bailey, J: "Historical Origin and Stylistic Developments of the Five-String Banjo", p. 59, The Journal of American Folklore, 1972.
- Winans, Robert; Gibson, George (2018). "Black Banjo, Fiddle and Dance in Kentucky and the Amalgamation of African American and Anglo-American Folk Music". Banjo Roots and Branches. Urbana: University of Illinois Press. p. 246.
- Bailey, J
- 23–26 December 1839. New York Herald. Quoted in Cockrell 52.
- Epstein, Dena (1977). Sinful Tunes and Spirituals: Black Folk Music to the Civil War. Urbana: Board of Trustees of the University of Illinois. p. 34. ISBN 0-252-00520-1.
- 18 March 1841. Playbill, Bowery Theatre. Quoted in Cockrell 148.
- 9 December 1841. Boston Post. Quoted in Cockrell 148.
- 31 January 1842. New York Herald. Quoted in Cockrell 149.
- Carlin, Bob. The Birth of the Banjo: Joel Walker Sweeney and Early Minstrelsy, (2007): 153.
- Sam Sweeney Memorial
- Cockrell, Dale (1997). Demons of Disorder: Early Blackface Minstrels and Their World. Cambridge University Press.
- Nathan, Hans (1962). Dan Emmett and the Rise of Early Negro Minstrelsy. Norman: University of Oklahoma Press.
- Carlin, Bob (2007). Birth of the Banjo: Joel Walker Sweeney and Early Minstrelsy. Jefferson, NC and London: McFarland & Co.
- Rice, Edward LeRoy (1911). Monarchs of Minstrelsy: From "Daddy" Rice to Date. New York: Kenny Publishing.
- Bailey, Jay (1972). "Historical Origin and Stylistic Developments of the Five-String Banjo". "The Journal of American Folklore", 85(335).