Mesa was initially signed as an amateur free agent by the Toronto Blue Jays in 1981, originally as an outfielder. He was traded to the Baltimore Orioles, during the 1987 season for starting pitcherMike Flanagan. Mesa began his major-league career as a starting pitcher with the Orioles, who hoped that Mesa's superior fastball would earn him success in the starting role. After 48 starts of below-average results during four seasons, the Orioles traded Mesa to the Cleveland Indians in 1992. Cleveland continued to use Mesa as a starter through the end of 1993; that year, Mesa pitched a career-high 208-2/3 innings, although his earned-run average was worse than the league average ERA for the fifth consecutive season.
Mesa became a relief pitcher for the Cleveland Indians in 1994, and for the first season of his career, he posted an ERA better than the league average. In the role of closer during the 1995 season, Mesa pitched superbly; in 64 innings pitched over 62 appearances, Mesa had a 1.12 ERA, saved 46 games, and won three. Thirty-eight of Mesa's 46 saves were recorded in consecutive appearances in save situations; this was a major-league record at the time. Mesa's performance in 1995 was instrumental in the Indians' 100–44 regular-season record and their first World Series appearance since 1954. In 1997, Mesa's 2.40 ERA, 16 saves, and four wins helped the Indians to their second World Series appearance in three seasons; however, he failed to hold a one-run lead in the ninth inning of Game 7 of the 1997 World Series, and Cleveland traded him mid-way through the following season. Following his departure from Cleveland, he pitched for the San Francisco Giants and Seattle Mariners, but without much success. In 2001, he signed with the Philadelphia Phillies and enjoyed a two-year run of success. However, in 2003 his ERA more than doubled, and he was granted free agency at the end of the season. He spent 2004 and 2005 pitching for the Pittsburgh Pirates, and pitched for the Colorado Rockies in 2006.
On December 10, 2006, he signed with the Detroit Tigers. He was released by the team on June 3, 2007. In 16 appearances with the team, he was 1-1 with a 12.34 ERA. On June 8, 2007 he signed a minor league deal with the Phillies. He appeared in 40 games for the Phillies, compiling a 1-2 record with a 5.54 ERA, and retired following the 2007 season.
Mesa was involved in a longstanding feud with former teammate Omar Vizquel following the publication of Vizquel's autobiography, Omar! My Life On and Off the Field. In the book, Vizquel criticized Mesa's performance in Game 7 of the 1997 World Series: "The eyes of the world were focused on every move we made. Unfortunately, Jose's own eyes were vacant. Completely empty. Nobody home. You could almost see right through him. Not long after I looked into his vacant eyes, he blew the save and the Marlins tied the game." Mesa reacted furiously, pledging to hit Vizquel upon every subsequent opportunity: "Even my little boy told me to get him. If I face him 10 more times, I'll hit him 10 times. I want to kill him." By the end of the 2007 season, Mesa had hit Vizquel twice.
Mesa was charged with one count of rape for allegedly penetrating one woman with his finger and two counts of gross sexual imposition for allegedly groping two women in a Lakewood, Ohio motel room on December 22, 1996. Mesa was acquitted of all charges on April 9, 1997.