Ruffin is a 1982 graduate of J. M. Hanks High School. Ruffin later attended the University of Texas where he was a member of the 1983 National Champional Squad. He was selected in the second round of the 1985 Major League Baseball Draft. He finished 7th in voting for the 1986 National League Rookie of the Year with a 9-4 record and a 2.46 ERA. He was promoted from the minor leagues to replace Steve Carlton in the Phillies' rotation after Carlton's release.