He was born in Williamston, North Carolina and attended Campbell University until being signed by the Indians in 1956. He is the older brother of Hall of Fame pitcher Gaylord Perry and was a fine pitcher in his own right. The Perry brothers trail only the Niekro brothers (Phil and Joe) for career victories by brothers. In 1959, Jim Perry came in 2nd to Bob Allison in the Rookie of the Year vote. Perry followed up with an 18-win season in 1960. His tenure with the Indians is highlighted in the book "Portrait of a Franchise: An Intimate Look at Cleveland Indians Baseball During the Rockin' Sixties" by Doug Kurkul.
Perry was a three-time All-Star and won the 1970 AL Cy Young Award, when he posted a record of 24-12. Jim and Gaylord Perry are the only brothers in Major League history to win Cy Young Awards. He also won 20 games in 1969, and won at least 17 games five times. As a batter, Perry was a switch-hitter and posted a respectable .199 batting average in his career. On July 3, 1973, brothers Gaylord Perry (Indians) and Jim Perry (Tigers) pitched against each other for the only regular season game in their careers. Neither finished the game, but Gaylord was charged with the 5–4 loss. Two Norm Cashhome runs helped Detroit.
Following his final year with Oakland, Perry retired to his North Carolina home where he keeps busy with charitable events, especially golf tournaments. His son, Chris, is a professional golfer who has won a tournament on the PGA Tour.