Van Buren was born in La Ceiba, Honduras but, after he was orphaned as a boy, he was sent to live with relatives in New Orleans. From Warren Easton High School in New Orleans, he received an athletic scholarship to Louisiana State University, where he led the nation in points (110) and touchdowns (16) as a senior.
A first-round draft pick in the 1944 NFL Draft, Van Buren ran for 444 yards in nine games during his first season with the Philadelphia Eagles. He also led the league in returning punts as a rookie. The following season, Van Buren led the NFL in rushing for the first time. He would win three additional rushing crowns in 1947, 1948 and 1949, becoming the first running back in NFL history to achieve three consecutive rushing titles. In 1948, he scored the 50th touchdown of the Eagles' season, setting the record for most touchdowns scored in a season. Jim Brown (1957–1961, 1963–1965) twice, Earl Campbell (1978–1980), and Emmitt Smith (1991–1993) have since managed that feat. All four are in the Pro Football Hall Of Fame.
Van Buren's signature game came on December 19, 1948. Playing in a blizzard for the NFL Championship against the Chicago Cardinals, Van Buren scored the only touchdown of the game to give the Eagles their first league title. They would win their second crown a year later against the Rams. In that game, Van Buren set a league record with 196 yards rushing.
The greatest irony was that Van Buren almost missed the game. Thinking the game wouldn't be played in the blizzard, he remained home until Eagles coach Earle "Greasy" Neale called him and told him the game was still on. He had to catch 3 trolleys and walk 6 blocks in order to make the game on time.
After a knee injury before the 1952 season, Van Buren retired as the NFL record holder for career rushing yards (5860) and career rushing touchdowns (69). He broke Clarke Hinkle's record of 3860 yards on October 3, 1949 when he rushed for 135 yards on 33 attempts and scoring two touchdowns in a 22-14 win over the Detroit Lions. He also scored three times returning kickoffs, three times on receptions, and twice on punt returns for a total of 77 touchdowns. He was also the first running back to pass 1000 yards in a season twice. A fast, powerful back, he stood 6-1 and weighed 205 pounds.