He was named as the head men's basketball coach at Ohio University on June 27, 2008, replacing Tim O'Shea, who resigned to take the head coach position at Bryant University. Groce was at Ohio from 2008 to 2012 and led the school to the NCAA Tournament twice. Qualifying as a #14 seed in 2010, Groce led the Bobcats to an upset win over Georgetown in the first round. In 2012, Groce led the #13 seed Bobcats all the way to the Sweet Sixteen, where they suffered a 73–65 overtime loss to North Carolina. Ohio had not been that far in the tournament since 1964. In four seasons at Ohio, Groce was 85–56 overall and 34–30 in Mid-American Conference games.
Groce finalized negotiations to become the men's basketball head coach at the University of Illinois on March 28, 2012. The Illini started 12-0 in 2012 under Groce, the best start for a first year coach in the team's modern era. The team won the 2012 Maui Invitational by defeating Butler University in the championship game. Groce's top priority at Illinois is getting his players to buy into and demonstrate the values of what he calls Toughness and Togetherness, or "T-n-T". Groce demands hard work and emphasizes the importance of every practice, game, and loose ball. To motivate his team, he had his players wear wristbands with the date "3-19-13" on it, which is the first day of the 2013 NCAA Tournament.
On December 21, 2013, John Groce ended four years of discontentment for Illinois fans by beating the University of Missouri, 65-64. In 2014, Groce continued Illinois' success in the month of November, improving to 21-0 under Groce and 32-0 overall during the past four seasons. Illinois is the only program in the nation with an undefeated November record dating back to 2011.
National champion Postseason invitational champion Conference regular season champion Conference regular season and conference tournament champion Division regular season champion Division regular season and conference tournament champion Conference tournament champion
^pg. 36 of the Summer 2007 edition of Taylor: A Magazine for Taylor University Alumni and Friends ISSN 1073-4376 "Coach Patterson used to tell me be the 'Big time' wherever you are...I'm not the guy who chases jobs. I feel fortunate to be at Ohio State. I love being here. We want to be the best we can every day."