After his sportscasting days Simpson retired to St. Croix, Virgin Islands. Among other firsts he was the initial U.S. sportscaster to appear live via satellite from Asia, and he was involved in the first American sportscast using instant replay technology. In 2005, ESPN brought Simpson back from retirement to do play-by-play for a series of college basketball games in a "turn back the clock" format on the ESPN Classic network.
Although he covered many sports, Simpson favored football and basketball. In his book Voices of the Game (1987), author Curt Smith quotes Simpson on the subject: "You do basketball and football, and their action carries you along. You tell what's happening; you don't have to hype. Golf is tougher—so many dead spots. But baseball! It's even harder. You have a whole season of dead time between pitches—what do you do with it? It takes a real artist to fill in, to sort of ride the game's stream along." This led Smith to comment, "A baseball artist Simpson was not," and even describe him as "chillingly indifferent" to baseball.
In his book Living a Dream (2003) Dick Vitale tells how Simpson taught him an important lesson about sportscasting: "One night, a long time ago, [Simpson] and I were doing a game—a so-so game. Somebody came up to me and asked, 'What game do you guys have coming up?' I turned to the guy and said, 'Aw, it's just another game, man, just another game.' Jim grabbed me and said, 'There is no such thing as just another game.' He was right."
Asked at his NSSA Hall of Fame induction ceremony about changes in sports over the last forty years, Simpson offered a rueful, humorous response: "I look around at different ballparks and I haven't been in some of them. Used to be, no matter where they were, I had been there."
During the final illness of his friend and colleague Bud Wilkinson, Simpson commented: "Bud is the only man I know that when he sees you or leaves you has the capability to throw his arms around you, with tears in his eyes, and say to another man, 'Do you know how much I love you?' That's not supposed to be the macho thing to do, but this guy doesn't have to prove anything as a football player, as a coach or as a man."