Baxter prepped at Mission Bay Senior High School in San Diego, California, where he played quarterback, running back and wide receiver as a senior. Mission Bay went 13-0 and won the CIF San Diego Division IV title in 2009. Baxter won the prestigious Silver Pigskin Trophy, awarded annually to San Diego County's most outstanding football player, chosen by KUSI television's Prep Pigskin Report. In addition, Baxter was a consensus High School All-American and also won the Hall Trophy. He joined the University of Southern California in 2010. In November 2010, Baxter was ruled temporarily ineligible for accepting benefits from an NFLPA-certified agent, but was later reinstated.
On October 25, 2011, ESPN reported that Baxter had been taken off the USC football team, but was still enrolled at the university. Baxter then enrolled at San Diego State University and was scheduled to miss the 2012 season due to NCAA transfer restrictions. On, February 29, 2012, however, Baxter has been dismissed from the SDSU football program for "various reasons".