Jolley attended Brigham Young University, where he was a first-team All-Mountain West selection in 2001. He was selected by the Oakland Raiders in the second round of the 2002 NFL Draft, and was a starter on the Raiders' Super Bowl team that year. He caught a touchdown pass in the AFC Championship Game, and hauled five more catches in the Super Bowl. He was the first St. George native ever to start a Super Bowl game.
In 2005, Jolley was dealt from the Raiders to the Jets, with a 2nd round pick and two 6th round picks, for a 1st and 7th round pick. After a year in New York, he was traded from the Jets to Tampa Bay for a sixth round draft pick.