Steinberg played with Soul Coughing throughout the band's entire history, from 1992 to 2000. In 2001, Steinberg performed on stage with Neil Finn and Tim Finn, Eddie Vedder, Johnny Marr, Ed O'Brien, Phil Selway and others in a series of concerts recorded on a DVD titled "7 Worlds Collide". He also recorded and performed with guitarist Marc Ribot and appeared on William Shatner's 2004 album Has Been. Steinberg performed on the Dixie Chicks' 2006 Accidents and Accusations tour, and toured with k.d. lang during the summer of 2007. Steinberg performed with Vanessa Carlton on The Tonight Show with Jay Leno October 11, 2007. In 2009, he appeared again with Neil Finn and friends for the second installment of 7 Worlds Collide playing alongside members of Radiohead and Wilco. Steinberg made a brief appearance in the 2009 film Funny People, as the bassist in a band that Adam Sandler's character, a movie star, hired to jam with him.
Steinberg works as a session musician, playing electric bass and double bass on numerous albums. He contributed the bass tracks to the 2012 Fiona Apple album, The Idler Wheel..., and he toured with Fiona Apple as part of her backing band in 2012. In 2014, Steinberg also played bass supporting guitar virtuoso and producer Blake Mills, a tour where Fiona Apple made sporadic appearances, to promote Mills' second full-length album, Heigh Ho. Steinberg has also been a regular performer with the Watkins Family Hour, and the band released their debut album in July 2015. Steinberg has also worked with singer songwriter J.S. Ondara and has toured as a bassist for Iron & Wine. In 2020, he appeared on Fiona Apple's album Fetch the Bolt Cutters.
- "Radiohead, Johnny Marr, Neil Finn team up for charity album, gigs". NME. November 17, 2008. Retrieved 30 March 2011.
- "Blake Mills at Mayne Stage 9/28/2014". Music-defined.com. Retrieved January 18, 2020.
- Kaye, Ben. "Fiona Apple Details New Album, Fetch the Bolt Cutters, in Expansive Interview". Consequence of Sound. Retrieved 17 April 2020.