Born in St. Louis, Missouri, to a wealthy manufacturer of glassware, Drey began acquiring timberland in the Missouri Ozarks for reforestation and conservation in 1950. His holdings, much acquired for the price of back taxes, eventually grew to nearly 160,000 acres (650 km2), the largest private landholding in the state and larger than Missouri's entire state park system. The project, known as Pioneer Forest, is a commercial forest managed in the public interest, with single tree selection harvesting techniques.
Drey founded the L-A-D Foundation, which acquires and protects other natural areas in the state, leasing many of them to the state park system at $1 per year.
In 2005, Drey was placed at No. 6 on Slate magazine's annual list of the top 60 U.S. philanthropists, thanks to his gift of 146,000 acres (590 km2) of Ozark land, valued at $180 million, to the L-A-D Foundation.
Other Drey beneficiaries have included his alma mater Antioch College; John Burroughs School, which uses Drey land for biology and outdoor education courses; the Government Accountability Project; and Coalition for the Environment, Missouri's first independent citizens' group to address a broad range of environmental issues.
Drey is a 1935 graduate of John Burroughs School and a 1939 graduate of Antioch College. In 1991, he donated his papers to the Western Historical Manuscript Collection at the University of Missouri–St. Louis.
According to press reports, his answering machine message says, "I’m out planting a forest. Please leave your name and number and I’ll try to get back to you before it matures."