He was born in Wilsonville, in Boyle County, Kentucky, near Parksville. Wilsonville was a small settlement where newly freed slaves had lived. Marshall lived there until he was 5, moving to Columbus, Ohio; however, he came back to Wilsonville every summer until he was 15, when his grandfather died. His aunt, Ella Mae Marshall, was the first special education teacher in Boyle County, and she worked to preserve the black school where she and Marshall's mother had taught.
He played in Pro Bowls after the 1968 and 1969 NFL seasons. He recovered 30 fumbles, an NFL record. He was a member of the Vikings' famous "Purple People Eaters" (which consisted of Marshall (DE), Alan Page (DT), Gary Larsen (DT), and Carl Eller (DE), and was the final player from Minnesota's initial expansion team of 1961 to retire. The Vikings credit Marshall with 127 career quarterback sacks, second most in Viking History behind Eller. He is one of 11 players to have played in all four of the Vikings Super Bowl appearances in the 1970s.
During his time with the Minnesota Vikings, Marshall was involved in what is considered by many, including SI.com author John Rolfe, to be one of the most embarrassing moments in professional sports history. On October 25, 1964, in a game against the San Francisco 49ers, Marshall recovered a fumble and ran 66 yards with it the wrong way into his own end zone. Thinking that he had scored a touchdown for the Vikings, Marshall then threw the ball away in celebration. The ball landed out of bounds, resulting in a safety for the 49ers. According to Marshall, when he approached Vikings head coach Norm Van Brocklin afterwards, Van Brocklin, after a pause, said, "Jim, you did the most interesting thing in this game today." Despite the gaffe, the Vikings won the game 27–22, with the final margin of victory provided by a Carl Eller touchdown return of a fumble caused by a Marshall sack. Marshall later received a letter from Roy Riegels, infamous for a wrong-way run in the 1929 Rose Bowl, stating, "Welcome to the club".