Rumaker graduated from Black Mountain College in 1955 and later wrote a memoir of his time there. He hitchhiked to San Francisco where he encountered the literature of the Beat Generation. Returning to New York, he attended Columbia University and received an MFA in 1971, after which he began teaching writing.
Rumaker's first collection was in 1959 in the new-writer showcase Short Story 2, which featured short stories from him and three other authors, including Gertrude Friedberg. His short stories in the book received a favourable review in The New York Times, where he was described as an "impressive young writer.
His first book, The Butterfly, is a fictionalized memoir of his brief affair with a young Yoko Ono, published before Ono became famous. His short stories, Gringos and other stories, appeared in 1967. A revised and expanded version appeared in 1991. He began to write directly about his life as a gay man in the volumes A Day and a Night at the Baths (1979) and My First Satyrnalia (1981). The novel Pagan Days (1991) is told from the perspective of an eight-year-old boy struggling to understand his gay self. Black Mountain Days, a memoir of his time at Black Mountain College, has a strong autobiographical element. In addition, there are portraits of many students and faculty (including the poets Robert Creeley, Charles Olson and Jonathan Williams) during its last years, 1952-1956.
Following his graduation from Black Mountain College, Rumaker made his way to the post-Howl, pre-Stonewall riots gay literary milieu of San Francisco, where he entered the circle of Robert Duncan. His account of that time in the book Robert Duncan in San Francisco, first published by Donald Allen at his Grey Fox Press, gives an unvarnished look at the premier poet of the San Francisco Renaissance. Rumaker will release previously unpublished letters between himself and Robert Duncan for a new edition, published by City Lights.
- The Butterfly: A Story in Nine Parts (1962) Charles Scribner's Sons; (1968) Macdonald
- The Bar (1964) by Four Seasons Foundation
- "Exit 3" (1966) by Penguin Books
- Gringos and Other Stories (1967) by Grove Press
- Schwul (Queers) (1970) by März Verlag
- Robert Duncan in San Francisco (1976) by Grey Fox Press
- A Day and a Night at the Baths (1979) by Grey Fox Press (reprinted by Triton Books, 2010)
- My First Satyrnalia (1981) by Grey Fox Press, ISBN 0912516518
- To Kill a Cardinal (1992) by Arthur Mann Kaye, ISBN 0963296221
- Gringos and Other Stories (1997) by North Carolina Wesleyan College Press, ISBN 0933598262
- Pagan Days (1998) by Circumstantial Productions, ISBN 1891592106
- Black Mountain Days (2003) by Black Mountain Press, ISBN 0964902087 (reprinted by Spuyten Duyvil 2012)
- An Immodest Proposal (2004) by The Phi Press
- Pizza: Selected Poems (2005) by Circumstantial Productions, ISBN 1891592092
- Selected Letters of Michael Rumaker (2012) by The CUNY Poetics Document Initiative: Lost & Found, City University of New York, Center for the Humanities
- Robert Duncan in San Francisco: Expanded Edition (2013) by City Lights Publishers, ISBN 0872865908