Both of Lee's parents, Elton Lee and Anfernee Williams, are deaf; he communicates with them via sign language. Lee had a difficult childhood. His father was not regularly involved in his life. Lee spent the first twelve years of his life moving repeatedly between his mother and grandparents, both in low-income homes. The frequent moves forced him to repeat a grade early in elementary school. While Lee finished sixth grade, his grandfather died and his grandmother moved to the housing projects in Baldwin Village, Los Angeles; Lee and his younger sister chose not to move with her and instead became wards of the state. His two older brothers were involved in gangs: one, Terreal Reid, was murdered in a gang-related killing and the other, Donte Reid, was imprisoned in Arizona on a count of attempted murder. Lee previously tried to join the same gang, but his brothers prevented it. His sister still lives in the Los Angeles area.
When Lee started high school at Morningside High School, in Inglewood, California, he was living in a foster home. Lee played on an Amateur Athletic Union basketball team the summer after his freshman year of high school and became friends with Steven Hester Jr., a high school student from Inglewood attending the private Junípero Serra High School in Gardena, California. The Hester family liked Lee and, in September 2008, he moved in with them and began attending Serra. The family wanted to help Lee's younger sister, too, but were unable because of space and financial constraints. The Hesters became heavily involved in his life and continue to attend most of his games and practices at USC.
In 2011, as a true freshman, Lee earned a starting job at wide receiver across from his former high school teammate Robert Woods. He finished the season with 73 receptions for 1,143 yards and 11 touchdowns; due to sanctions, the 10-2 Trojans were prohibited from playing in either the inaugural Pac-12 Conference Championship game or a post-season bowl game. The combined receptions by Lee and Woods (184) and receiving yards (2,435) in 2011 were the most by a pair of Trojans in a season; Lee was awarded the 2011 Pac-12 Freshman Offensive Co-Player of the Year and named to the All-Pac-12 second team.
Against Arizona in 2012, Lee set the Pac-12 Conference record for receiving yards in a game with 16 receptions for 345 yards and two touchdowns. The following week, Lee had 251 return yards against Oregon, setting another conference record. He finished the season with 118 catches for 1,721 yards and 14 touchdowns and was named the 2012 Pac-12 Conference offensive player of the year. Lee won the 2012 Fred Biletnikoff Award as the top wide receiver in the nation, becoming the first Trojan to win the award. He was also a unanimous All-American.
As a junior in 2013, Lee played in 11 games, recording 57 receptions for 791 yards and four touchdowns. On January 3, 2014, Lee announced that he would forgo his senior season and enter the 2014 NFL Draft. During his three-year career he had 248 receptions for 3,655 yards and 29 touchdowns.