Woolsey was born on the Breeze Hill plantation in Aiken, South Carolina, as Elizabeth (Elsa) Gammell Woolsey, but in later years took her middle name, which she shortened to "Gamel" (a Norse word meaning "old"). Her father was plantation owner William Walton Woolsey (1842–1909). The Woolsey branch of the New England Dwight family had influence in the law, the church and education. He had first married Catherine Buckingham Convers, daughter of Charles Cleveland Convers, but she died in 1888. Her aunt, Sarah Chauncey Woolsey – better known by her pen name, Susan Coolidge – wrote the popular Katy series and other children's fiction. Her relations included three presidents of Yale University.
Her mother was her father's second wife, Bessie Gammell, daughter of William A. Gammell. Her mother traced her maternal ancestry to William Washington. After her father's death in 1910 they moved to Charleston, where she went to day school. Despite weak health following an attack of tuberculosis in 1915, she left home for New York in about 1921, hoping to be an actress or a writer. Her first known published poem appeared in the New York Evening Post in 1922. The following year she met and married Rex Hunter, a writer and journalist from New Zealand, but they separated after four years. In 1927, while living in Patchin Place, Greenwich Village, she met the writer John Cowper Powys and, through him, his brother Llewelyn and his wife, Alyse Gregory. She and Alyse became friends for life, while with Llewelyn she had a passionate and painful love affair.
She left New York for England in 1929, settling in Dorset to be near Llewelyn, where she came to know the whole Powys family and their circle. Parting from Llewelyn in 1930, she married the historian and writer Gerald Brenan in a private ceremony, and they lived together, mainly in Spain, until her death. In 1933 she began an enduring friendship with Bertrand Russell, who wanted to marry her.
Woolsey, primarily a poet, published very little in her lifetime: Middle Earth, a collection of 36 poems, came out in 1931, Death's Other Kingdom in 1939 (re-released as "Malaga Burning" in 1998 by Pythia Press) and Spanish Fairy Stories in 1944. Her Collected Poems have been published since her death. Patterns on the Sand (published by The Sundial Press in 2012) recalls her South Carolina childhood; One Way of Love, accepted by Gollancz in 1930 but suppressed at the last minute because of its sexual explicitness, was published by Virago Press in 1987. She died in Spain in 1968 of cancer, and is buried at the English Cemetery, Malaga.
- Benjamin Woodbridge Dwight (1874). The history of the descendants of John Dwight, of Dedham, Mass 1. J. F. Trow & son, printers and bookbinders. p. 288.
- Second Supplement to the History of the class of 1864. Yale College. 1895. p. 49.
- Kenneth Hopkins (Summer 1985). "Bertrand Russell and Gamel Woolsey". Russell: the Journal of Bertrand Russell Studies (McMaster University): 50–58.
- The Queen of Spain's Literary Past The Olive Press, October 15, 2007. Retrieved February 11, 2013.
- Eland Books Specialists in travel literature and publishers of Death's Other Kingdom