Ballard was a one year starter at Wilmington College then transferred after his freshman year to Mount Union College. While at Mount Union, Ballard broke numerous school, conference and NCAA records en route to leading the Purple Raiders to their first ever Division III National Championship in 1993. A two time First Team All-American Team member in 1992 and 1993, and an Honorable Mention All-American in 1991, Jim shattered 17 Division III records and threw for over 12,000 yards and over 150 touchdowns. The two-time recipient of the Mike Gregory Award, which is given to the Ohio Athletic Conference's top offensive back, he was a three time All-Conference selection and suffered only one OAC loss during his college career. Ballard won the inaugural Melberger Award and the Gagliardi Trophy as Division III's Player of the Year. He held every Mount Union passing record by the end of his career and still holds the NCAA record for highest number of touchdowns thrown in a playoff game (8). In addition, Jim was selected as the 1993 Joe Fogg Award Winner, given to the top collegiate player in all divisions in the Midwest by the Cleveland Touchdown Club. In 2008, Jim was inducted into the College Football Hall of Fame. Ballard was an All-American at Mount Union College.
As a coach, Ballard has experience at Mount Union College as QB coach in 1994-95, Alliance High School as QB coach in 1996 and 1998, Cuyahoga Falls High School as QB coach in 2001-03, and Assistant Head Coach of the Canton Legends of the American Indoor Football Association in 2005. He was hired in 2013 to be an Elite 11 Coach and was also offered the NCAA Football All-Star Game Head Coach position. In May 2013, USA College Football announced that the Division III Quarterback of the Year Award would be renamed The Jim Ballard Trophy.
Ballard is an analyst for Mount Union football games on Sports Time Ohio with Joe Tait and Harry Paidis as well as being a regular guest on Big Time Sports on AM 1350.