He was named as the head men's basketball coach at Ohio University on June 27, 2008, replacing Tim O'Shea, who resigned to take the head coach position at Bryant University. Groce was at Ohio from 2008 to 2012 and led the school to the NCAA Tournament twice. Qualifying as a #14 seed in 2010, Groce led the Bobcats to an upset win over Georgetown in the first round. In 2012, Groce led the #13 seed Bobcats all the way to the Sweet Sixteen, where they suffered a 73–65 overtime loss to North Carolina. Ohio had not been that far in the tournament since 1964. In four seasons at Ohio, Groce was 85–56 overall and 34–30 in Mid-American Conference games.
Groce finalized negotiations to become the men's basketball head coach at the University of Illinois on March 28, 2012. The Illini started 12-0 in 2012 under Groce, the best start for a first year coach in the team's modern era. The team won the 2012 Maui Invitational by defeating Butler University in the championship game. Since 2013, John Groce has reversed Illinois' prior, four-game losing streak in the annual Braggin' Rights interstate rivalry against the University of Missouri by winning the contest for three straight seasons. During their 2014 season, Illinois' success in the month of November improved to 21-0 under Groce and 32-0 overall since 2010. Until losing their first non-conference game in 2015, Illinois was the only program in the nation with an undefeated November record dating back to 2011.
Groce's stated top priority at Illinois is getting his players to buy into and demonstrate the values of what he calls Toughness and Togetherness, or "T-n-T". To motivate his team during the 2013 season, he had his players wear wristbands that read "3-19-13"—the starting date of the 2013 NCAA Tournament.
National champion Postseason invitational champion Conference regular season champion Conference regular season and conference tournament champion Division regular season champion Division regular season and conference tournament champion Conference tournament champion
^pg. 36 of the Summer 2007 edition of Taylor: A Magazine for Taylor University Alumni and Friends ISSN 1073-4376 "Coach Patterson used to tell me be the 'Big time' wherever you are...I'm not the guy who chases jobs. I feel fortunate to be at Ohio State. I love being here. We want to be the best we can every day."