Carlesimo is a 1971 graduate of Fordham University, where he played basketball under coach Digger Phelps. He began his coaching career as an assistant at the school after he graduated. He received his first head coaching job when he took over New Hampshire College for one year, beginning in 1975. The New Hampshire Penmen ended the 1975–1976 season with a 14-13 record and won the Mayflower Conference championship. Carlesimo also had a successful coaching stint at Wagner College of Staten Island, New York, a Division I school, leading the team to two NIT berths in six years as coach from 1976 to 1982. In addition, Carlesimo spent time coaching in Puerto Rico during the summers.
Carlesimo coached Seton Hall University from 1982–1994, leading the turnaround of a struggling program. The Pirates made their first ever NCAA tournament in 1988, and Carlesimo was named the Big East Conference coach of the year.
In 1989, Carlesimo led the Pirates to their first-ever Final Four appearance following an 78-65 upset over the #2 seed Indiana Hoosiers coached by Bobby Knight. They advanced to the NCAA Championship game against the Michigan Wolverines. In a controversial overtime, the Pirates lost 80-79. Carlesimo was again named Big East coach of the year, following the 1989 season.
During Carlesimo's tenure, the Pirates made the NCAA tournament six times, four consecutively from 1991 until 1994. After going 6-23 his first season, he turned the team into a consistent title contender and eventual tournament runner-up and number one seed. Compiling an overall record of 212–166, he was named Seton Hall's "Coach of the Century."
On June 23, 1994, Carlesimo was hired as head coach of the Portland Trail Blazers. Midway into his first season with the team, the team traded long-time Blazer Clyde Drexler in an effort to rebuild. However instead of falling in the standings, Carlesimo led the team to a winning record and the playoffs in his first season as an NBA head coach. He led the Blazers to the playoffs in each of his three years as their head coach, but was unable to lead the team out of the first round of the playoffs and was fired following the 1996–97 season.
In 1997, Carlesimo headed to the Golden State Warriors, again succeeding Rick Adelman as head coach (Adelman had previously coached the Trail Blazers prior to Carlesimo's hiring). He coached that team until December 1999, at which point he was fired after his team got off to a losing start for the third straight year.
In a practice during the 1997-98 season while Carlesimo was head coach of the Warriors, Carlesimo was attacked by Latrell Sprewell. Sprewell choked Carlesimo for nearly 15 seconds until being pulled off by several teammates, and elbowed him several minutes later. The Warriors terminated Sprewell's contract, and the NBA suspended him for one year (later reduced to 68 games by an arbitrator); he never played for the Warriors again and continued to have troubles on and off the court.
After leaving the Warriors, he worked as a broadcaster and analyst with The NBA on TNT until being hired as an assistant coach under Greg Popovich in 2002. He worked with the Spurs until 2007, winning three championships in five seasons (2003, 2005 and 2007).
On December 27, 2012, he was promoted to interim head coach of the Nets after the firing of Avery Johnson. Carlesimo took a team floundering at 14-14, and compiled 35-19 record for a fourth-place finish in the East, earning home-court advantage for the Nets in the first round. On May 5, 2013, Nets GM Billy King announced Carlesimo would not be retained as head coach after the team lost in their first round series against the Chicago Bulls in 7 games.