The eldest of nine children, Frazier attended Atlanta's David Tobias Howard High School. He quarterbacked the football team and played catcher on the baseball team. He learned basketball on a rutted and dirt playground, the only facility available at his all-black school in the racially segregated South of the 1950s. After Howard, Frazier attended Southern Illinois University. Although he was offered other scholarships for his football skills, Frazier accepted a basketball offer from Southern Illinois University.
Frazier became one of the premier collegiate basketball players in the country. He was named a Division IIAll-American in 1964 and 1965. In 1965, Frazier led SIU to the NCAA Division II Tournament only to lose in the finals to Jerry Sloan and the Evansville Purple Aces 85-82 in overtime. In 1966, he was academically ineligible for basketball.
In 1971, the New York Knicks traded for fellow NBA guard Earl "the Pearl" Monroe to form what was known as the "Rolls Royce Backcourt" with Frazier. While there were initial questions as to whether Frazier and Monroe could coexist as teammates, the duo eventually meshed to become one of the most effective guard combinations of all time, leading the Knicks to the 1973 NBA championship. That pairing is one of few backcourts ever to feature two Hall of Famers and NBA 50th Anniversary Team members.
Frazier held Knicks franchise records for most games (759), minutes played (28,995), field goals attempted (11,669), field goals made (5,736), free throws attempted (4,017), free throws made (3,145), assists (4,791) and points (14,617). CenterPatrick Ewing would eventually break most of those records, but Frazier's assists record still stands.