In 1938 Graf enrolled at Ohio State, where his father Campbell "Honus" Graf had been a three-sport athlete and the 1914 football captain. After college Honus had been a graduate assistant on the 1915 team and played professional football with Peggy Parratt's Cleveland Indians football club. He later served on the Ohio State Athletic Council.
Jack Graf joined the Ohio State football team as quarterback and served as a backup to All AmericanDon Scott for two years. In 1941, before Graf's senior year, Paul Brown replaced Francis Schmidt as an Ohio State head coach and Brown moved Graf to fullback. As the featured back on the Ohio State offense, Graf led the Buckeyes to a 6-1-1 record. He won the Chicago Tribune Silver Football award, the Most Valuable Player award in the Big Ten. His senior year he ran for 10 touchdowns while passing for 2 more. Meanwhile, Graf was also a guard on the Ohio State basketball team and was named team captain as a senior.
The Cleveland Rams selected Graf in the 1942 NFL Draft, but he instead attended Harvard Business School. In 1945 Graf and his brother Campbell Jr. joined their father's electrical engineering company to form Graf and Sons. He also served 26 years as an assistant coach with the Ohio State basketball team.