Adderley was born in Philadelphia, Pennsylvania on June 8, 1939 to parents Charles and Rene (White). Adderley grew up there as well. He attended Northeast High School beginning in 1953. He played football, basketball, and baseball, and won All-City Honors in all three.
After high school, Adderley chose to attend Michigan State University under head coach Hugh "Duffy" Daugherty. Adderley led the team in rushing yards in 1959 and pass receptions in both 1959 and 1960. Adderley was the co-captain of the team in his senior year. In 1960, he made the All-Big Ten Conference team and played in the East-West Shrine Game, the Coaches' All-American, and the College All-Star games. Adderley was picked for the All-Michigan State University team in 1970.
Adderley was drafted by the Green Bay Packers in the first round of the 1961 NFL Draft with the 12th overall pick. He began his professional career as a halfback, but was later switched to defense because the Packers already had eventual Hall of Fame runners in Paul Hornung and Jim Taylor. Adderley was first moved to cornerback to replace an injured teammate. In 1962, the move became permanent and Adderley went on to become an all-NFL selection five times in the 1960s. Packers coach Vince Lombardi remarked, "I was too stubborn to switch him to defense until I had to. Now when I think of what Adderley means to our defense, it scares me to think of how I almost mishandled him."
Adderley recorded 39 interceptions in his nine seasons with the Packers. He holds the Green Bay records for interceptions returned for touchdowns in a career (seven, record tied with Darren Sharper), and interceptions returned for touchdowns in one season (three, in 1965).
Adderley started for the Packers from 1961–69, then played three seasons (1970–72) with the Dallas Cowboys. While with the Packers, he won rings in Super Bowl I and II. Adderley was a major factor in the Packers' Super Bowl II win over the Oakland Raiders, intercepting a pass by Raiders quarterback Daryle Lamonica in the fourth quarter and returning it 60 yards for a touchdown to put the game away. This was the first Super Bowl touchdown on an intercepted pass. After joining the Cowboys, Adderley became a vital cog in Dallas' "Doomsday Defense," assisting the Cowboys to a Super Bowl appearance in Super Bowl V and a championship win in Super Bowl VI.
Adderley was traded to the Los Angeles Rams in the summer of 1973 but instead of reporting to the team, he retired on August 7.
Along with two former Packer teammates - Fuzzy Thurston and Forrest Gregg - Adderley is one of only three players in pro football history to play on six World Championship teams. However, in a revised edition of Instant Replay, a memoir by former Packer teammate Jerry Kramer, Adderley is quoted as saying, "I'm the only man with a Dallas Cowboys Super Bowl ring who doesn't wear it. I'm a Green Bay Packer."
In his 12 NFL seasons, Adderley recorded 48 interceptions, which he returned for 1,046 yards and seven touchdowns, an average of 21.8 yards per return. He also recovered 14 fumbles (returning them for 65 yards) and returned 120 kickoffs for 3,080 yards and two scores.