Steinberg has periodically worked as a session musician, playing electric bass and double bass on numerous albums. He contributed the bass tracks to the 2012 Fiona Apple album, The Idler Wheel..., and he toured with Fiona Apple as part of her backing band in 2012. Steinberg has also been a regular performer with the Watkins Family Hour, and the band released their debut album in July 2015. He is currently touring with the Watkins Family Hour in 2015.
Steinberg made a brief appearance in the 2009 film Funny People, as the bassist in a band that Adam Sandler's character, a movie star, hired to jam with him.
Steinberg was born in New York City and grew up in Newton, Massachusetts, son (with brother Adam) of musicologist Michael Steinberg and fabric artist Jane Steinberg. He currently resides with two cats, Rupert and Little Charlie Onions, more commonly referred to as "House Bears".