Grossman was an All State performer at Haverford High School, in suburban Philadelphia, where from a young age he would reply to the question, "What will you do when you grow up?" with the certain retort: "I'm going to be a professional football player." He was a varsity letterman in both football and wrestling at Haverford Senior High School. His father was a butcher.
Grossman was an outstanding tight end (and three-year starter) for the Temple Owls in the early 1970s. In 1972, he caught 23 passes for 349 yards and four touchdowns and Temple had a record of 5–4–0. The following year, Grossman was named Associated PressAll-America third team and Temple notched a record of 9–1–0. That year, Grossman led the team in receptions with 39 for 683 yards and 4 touchdowns; Temple outscored its opponents 353-167. Grossman finished his career at Temple with 89 receptions for 1505 yards and 10 touchdowns. Grossman was also a member of Pi Lambda Phi Fraternity's Pa Alpha Delta Chapter.
Grossman joined the Steelers as an undrafted free agent in 1974, wearing number 84. He showed his skill when he caught a crucial touchdown pass from Terry Bradshaw early in Super Bowl X. In 1976, they drafted tight end Bennie Cunningham, behind whom Grossman played for two years, until the 1978 season when Cunningham was injured. Undersized and underweight, Grossman stepped up to the position and had a prolific season. In just 10 starts, Grossman had a career-high 37 receptions (the most by a Steeler tight end in 12 years) for 448 yards and 1 touchdown, and contributed to the Steelers Super Bowl XIII win. Grossman would play 3 more years with the Steelers before retiring, earning 4 Super Bowl rings for Super Bowl IX, Super Bowl X, Super Bowl XIII, and Super Bowl XIV. Grossman said that his nickname of "The Rabbi" referred to his Jewish ancestry and came from Dwight White, who was the authority for assigning such titles to the Steelers. He also said he never faced any anti-Semitism during his career, because “in sports -- in my era and currently -- it really is the great melting pot," he said. "If you ‘bring game,’ you’re fine. If you’re an imposter, then they’ll run you out regardless of what your religious preferences are or ethnic background is. It was obviously different in the ’60s, ’50s, ’40s, but from the time that I’ve been involved, it’s been completely open and purely performance-based acceptance or non-acceptance.”
Steelers President Dan Rooney said of Grossman: "Randy Grossman was one of those guys who was never viewed as one of the superstars, but he did everything you asked him. He caught the big touchdown pass in the Super Bowl. He was one of those guys who got you the first down when he would go in and play. He was just a real good competitor who really proved his value. He was just a good team man."
Steelers Director of Personnel Art Rooney, Jr., said that Grossman's hands were "the best", and that "he caught whatever was near him".