McCaffrey began his college career at Duke University where as a sophomore he was the team's second leading scorer during its 1991 NCAA Championship season. He scored 16 points in the 1991 title game win over Kansas. McCaffrey then sat out the 1991-92 season as he transferred to Vanderbilt University.
As a Commodore, McCaffrey was named a two-time All-American. A 6'3" shooting guard, he averaged 20.6 points in both his seasons at Vanderbilt, leading the school to a high national ranking of number 5, an SEC regular season title with a 14-2 record, and number 3 regional seed in the 1993 NCAA Men's Division I Basketball Tournament where Vanderbilt lost in the Sweet 16s to Temple. His 20.6 point average and .464 three-point field goal ranks second all-time in school history, and he is first in career free throw percentage at 88%. He also holds the Vanderbilt record for most assists in a game, 14, scored January 13 against Kentucky. He shared SEC Player of the Year honors with Jamal Mashburn in 1993.