Schwartz's books of poetry include Cairo Traffic (University of Chicago Press, 2000) and the chapbook Greatest Hits 1973-2000 (Pudding House Press, 2003), which were preceded by Goodnight, Gracie (1992) and These People (1981). He co-edited the collection Elizabeth Bishop and Her Art (University of Michigan Press, 1983). In 1990, he adapted These People for the Poets' Theatre in a production called These People: Voices for the Stage, which he also directed.
Schwartz served as co-editor of an edition of the collected works of Elizabeth Bishop for the Library of America, entitled Elizabeth Bishop: Poems, Prose, and Letters (2008) and edited the centennial edition of Elizabeth Bishop's Prose for Farrar, Straus and Giroux (2011).
His poems, articles, and reviews have appeared in The New Yorker,The Atlantic,Vanity Fair,The New Republic,The Paris Review,Ploughshares,Agni,The Pushcart Prize,The Best American Poetry, and The Best of the Best American Poetry. Between 1968 and 1982 he worked as an actor in the Harvard Dramatic Club, HARPO, The Pooh Players, Poly-Arts, and the NPR series The Spider's Web, playing such roles as Scrooge (A Christmas Carol), the Mock Turtle (Alice in Wonderland), Froth (Measure for Measure), Trofimov (The Cherry Orchard), Zeal-of-the-Land Busy (Bartholomew Fair), The Worm (In the Jungle of Cities), Krapp (Krapp's Last Tape), the Disciple John (Jesus: A Passion Play for Cambridge), and played a leading role in Russell Merritt's short satirical film The Drones Must Die. He also directed two operas, Ravel's L'Heure Espagnol (Boston Summer Opera Theatre) and Stravinsky's Mavra (New England Chamber Opera Group), 1972. He has appeared in The Poets' Theatre performances of Dylan Thomas's Under Milk Wood (2014) and The Word Exchange (2015).