Lofton graduated from Stanford University. As a senior in 1977, Lofton received 57 passes for 1,010 yards (17.72 yards per reception average) with 14 touchdowns, and was an AP & NEA Second Team All-American selection. He was a member of Theta Delta Chi Fraternity.
Lofton won the long jump at the 1978 NCAA Track and Field Championships with a wind-aided jump of 26 feet 11¾ inches. He won the long jump at the 1974 CIF California State Meet with a jump of 24 feet 3½ inches after placing sixth in this meet the year before. He was also a sprinter of note, with a best of 20.5 in the 200 meter. He has been an active participant in Masters track and field since 1997.
In his 16 NFL seasons, Lofton caught 764 passes for 14,004 yards and 75 touchdowns. He averaged 20 yards per catch or more in five seasons, leading the league in 1983 and 1984 with an average of 22.4 and 22 yards respectively. He also rushed 32 times for 246 yards and one touchdown.
Lofton is the first NFL player to record 14,000 yards receiving and was the second, one game after Drew Hill to score a touchdown in the 1970s, 1980s, and 1990s. During his nine seasons in Green Bay, Lofton played in seven Pro Bowls and left as the team's all time leading receiver with 9,656 yards (since broken by Donald Driver). In 1991, Lofton became the oldest player to record 1,000 receiving yards in a season (since broken by Jerry Rice). On October 21, of that same year, Lofton became the oldest player to record 200 yards receiving as well as 200 yards from scrimmage in a game (35 years, 108 days). He is also the 2nd oldest player to have 200+ all purpose yards in a game behind Mel Gray, (35 years, 204 days)
Lofton became the wide receiver coach for the San Diego Chargers in 2002 and continued that role until he was fired on January 22, 2008. In 2006, Lofton was one of two finalists for the Stanford head coaching job. That job went to Jim Harbaugh. Lofton was later announced as a candidate to become head coach for Oakland Raiders in 2007 but the job would later go to Lane Kiffin. In 2008, the Raiders hired him as their wide receivers coach. On January 13, 2009, Lofton was let go by the Oakland Raiders and replaced by Sanjay Lal.
In October 1984, a dancer at the Marquee Club in Milwaukee accused James Lofton and his Packers teammate Eddie Lee Ivery of sexual assault. Lofton and Ivery asserted that the acts were consensual. Neither player ended up being charged in the incident due to a lack of evidence. Two years later, Lofton was charged with second-degree sexual assault following an incident in the stairwell of a Green Bay nightclub. He was found not guilty of that charge.