Holt grew up in Gibsonville, North Carolina. He was Prep Football Report All-America selection, adding all-state honors at Eastern Guilford High School in Gibsonville. While there he caught 129 passes during his career, gaining 2,573 yards and scoring 42 touchdowns including 56 receptions for 983 yards and 17 touchdowns as senior. He also returned three punts and three kickoffs for touchdowns during his career. Additionally, Holt was a standout defensive back who posted 62 tackles and four interceptions as senior. He was named one of the Top 25 players in the state by the Charlotte Observer. After high school, Holt attended Hargrave Military Academy in 1994. There he caught 21 passes for 524 yards and six touchdowns. Torry also made an appearance with, at the time, St. Louis Rams teammate Marshall Faulk in Nelly's Air Force Ones Music Video.
Holt attended North Carolina State University, and played wide receiver for the NC State Wolfpack football team from 1995 to 1998. In his senior year Holt was named Atlantic Coast Conference Offensive Player of the Year. That season he set ACC records of 88 receptions (since broken by Kenneth Moore of Wake Forest) for 1,604 yards and an NC State record of 16 touchdown receptions. Holt was a consensus first-team All-American as senior. He was also a finalist for the Fred Biletnikoff Award, given to the nation’s top receiver. As a junior he led the team, setting Wolfpack season records with 62 receptions for 1,099 yards, topping marks of 55 by Naz Worthen (1988) while becoming the first player in team history to gain more than 1,000 yards in a season. He started in five of the first seven games as sophomore. He majored in sociology. Holt's number, 81, was retired in 1999.
Holt was the sixth overall draft pick in the 1999 NFL Draft and the first selection made by the St. Louis Rams. On July 23, 1999, Holt signed a five-year, $10 million contract, including a $5.4 million signing bonus, with the Rams. In his rookie season he posted 53 receptions, 788 total yards and six touchdowns on the way to the Super Bowl XXXIV championship. From 1999 to 2001, the Rams scored over 500 points each season, and their offense was dubbed "The Greatest Show on Turf".
Beginning in 2000 Holt reached at least 1,300 yards every season through 2005, a league record of six consecutive seasons. Holt's streak was broken in 2006, due to injuries to himself and other teammates that hindered the offense for parts of the year. Holt came into the NFL as number 88, but in 2001 changed his number to 81 (worn by Az-Zahir Hakim previously).
Holt's career also includes 7 Pro Bowls (2000, 2001, 2003–2007) including five straight, 74 career touchdowns for 448 points (including 2 two-point conversions) and 920 career receptions. He ranks among the top 10 active leaders in receiving yards, receiving touchdowns, and receptions, and has finished in the top ten of those three categories in five straight seasons(2003–2007). Holt has also led the league in receiving yardage on two separate occasions (2000, 2003), and receptions once (2003). Holt is tenth all time in receiving yards, and eleventh all time in pass receptions.
Prior to the 2003 seasons Holt agreed to a 7-year $42 million contract extension that included a $12.5 million signing bonus. Holt led the NFL in receptions in 2003 and led the NFL in receiving yardage in 2000 and 2003. He was a First-team All-Pro in 2003 and a Second-team selection in 2006. On October 15, 2006, Holt became the fastest player in NFL history to reach 10,000 receiving yards doing so in the sixth game of his eighth season and also to 11,000 yards.
His request for a release was granted by the Rams on March 13, 2009. If he was not released, he would have been due $5.65 and $6.65 million in the last two years of his contract and subsequently become a free agent in 2010.
Holt finished his 10-year career with the Rams starting 147 of 158 games, recording 869 receptions for 12,660 yards, and 74 touchdowns—ranking second in Rams' history in each category behind Isaac Bruce.
Holt was signed by the Jacksonville Jaguars on April 20, 2009, to a 3-year, $20 million deal. Holt had 51 catches, but for the first time in his career he did not have a touchdown reception. Holt was released on February 11, 2010, and only earned $3.45 million of the $20 million contract.
On April 20, 2010, Holt signed a one-year, $1.7 million contract with the New England Patriots. He was placed on injured reserve on August 15, 2010 as a result of a knee injury that would require surgery. He was released by the team on August 17, 2010 with an injury settlement.
On April 4, 2012, Holt signed a ceremonial contract with the St. Louis Rams to retire with the team. He retired ranked 10th in league history with 13,382 yards receiving and 13th with 920 receptions.
On November 20, 2010 during the Raycom Sports broadcast of the annual NCSU game with UNC-Chapel Hill, sideline reporter Mike Hogewood prefaced an interview with Holt stating that he had retired. Holt has since gotten into broadcasting with Fox Sports, providing commentary for their NFL coverage.
Holt joined NFL Network in 2010 as an analyst on the network’s signature show, NFL Total Access, as well as other NFL Network shows and specials. He made his debut as an analyst on the NFL Total Access: Pro Bowl Selection Show on Dec. 28, 2010.