Levy's family emigrated from Montreal, Quebec, Canada. His father, a decorated World War I veteran, ran a small business on the South Side of Chicago, Illinois. He graduated from South Shore High School in Chicago in 1943. Following graduation, he enlisted in the Army Air Forces and spent the remainder of World War II in the military; Levy was discharged from the army shortly after the war ended. Though he was known to use historical examples to inspire his teams, Levy corrected those who used war and combat metaphors to describe football games by telling them that he actually fought in a war and that football and war were in no way comparable. Referring to the Super Bowl, he once said "This is not a must-win; World War II was a must-win".
Levy's first coaching job was at St. Louis Country Day School, coaching football and basketball, the latter in which he coached to a championship. Two years later, Levy returned to Coe College as an assistant football coach (1953–1954) and in his second stint as a head coach, he also won a championship—once again in basketball. In 1954, he joined the coaching staff at the University of New Mexico and was named head coach in 1958. In two seasons as head coach, he guided the Lobos to a 14–6 record and earned Skyline Conference Coach of the Year honors both years. He interviewed with the University of California, Berkeley on February 2, 1960, and was announced as the new head coach of the Cal Bears on February 5, 1960. Despite selecting a young Bill Walsh as a coaching assistant, Levy's best record during his four season tenure as head coach at Cal from 1960 to 1963 was 4–5–1. He finished his college-coaching career with a five-year stint as head coach at the College of William & Mary where he twice earned Southern Conference Coach of the Year honors.
Midway through the 1986 season, following a two-year hiatus from coaching and one season as the head coach of the Chicago Blitz of the USFL, Levy returned to the NFL as head coach of the Buffalo Bills. He finished the season with a 2–5 record. In 1987, his first full season with the Bills, the team returned to respectability with a 7–8 record and were in the playoff hunt throughout most of the season. The following season the team posted a 12–4 record and won the first of six AFC Eastern Division titles. With his high-powered “no-huddle” offense, Levy's Bills went on to lead his AFC championship team to four consecutive Super Bowl appearances.
From 1988 through 1997, the Bills were first in the AFC in winning percentage and second only to the San Francisco 49ers in the NFL. Levy, the winningest coach in Bills’ history, recorded a 112–70 regular season record and was 11–8 in the playoffs during his eleven seasons with the Bills. He was named NFL Coach of the Year in 1988 and AFC Coach of the Year in 1988, 1993, and 1995.
Levy retired in 1997 and became an analyst for NFL.com. In 2001 Levy was elected to the Pro Football Hall of Fame in Canton, Ohio. Along with former Bills' special-teamer Steve Tasker, Levy did local broadcasts for the Bills' pre-season games until being appointed the Bills' general manager in 2006. During the regular season he was a part of the Chicago Bears pregame show on ESPN Radio 1000, as well as a Bears postgame show on Comcast SportsNet.
On January 5, 2006, Bills owner Ralph Wilson enlisted Levy, at the age of 80, to act as General Manager and Vice President of Football Operations for the Buffalo Bills. Following the resignation of Mike Mularkey, there was initial speculation (created by Levy's own comments at a team press conference) that Levy would resume a coaching role with the team. To eliminate this speculation, and to minimize any future tension between Levy and the Bills' new head coach, team owner Wilson said: "To say it very, very succinctly, Marv Levy is our general manager. He will never be the coach."
Levy's first order of business was to hire a new coach as a replacement for Mularkey, who resigned within days of Levy's appointment. After a strenuous interview process Levy and team owner Wilson hired Detroit Lions interim head coach Dick Jauron as coach. Jauron formerly was head coach of the Chicago Bears.
Following the Bills' last game of the 2007 season, Levy decided to step down as GM of the Bills (his two-year contract had expired). He has returned to live in his native Chicago, although he has also spent some time in Montreal mentoring then-Alouettes head coach Marc Trestman.
In 2009, Levy collaborated with Buffalo football historian Jeffrey Miller (Professional Football Researchers Association) to write a book entitled Game Changers: The Greatest Plays in Buffalo Bills Football History. In August 2011, Levy published a second book, Between the Lies, featuring a team based loosely on the Bills (including a quarterback named "Kelly James") progressing to the Super Bowl against a Los Angeles-based team and its take-no-prisoners head coach, while a scandal erupts, placing the integrity of the game at risk. Levy has indicated he has future books planned, some of which may not involve football.
Won two of three CFL championships in five seasons while head coach of the Montreal Alouettes
Guided the Bills to six division championships (including four consecutive from 1988–1991)
Compiled a 17–6 record (14–6 in the regular season and 3–0 in the post-season) against the winningest coach in NFL history, Don Shula. He is the only coach to have a winning record against Shula, other than Tom Flores of the Raiders, who went 6-1 against Don Shula.
Compiled 204 CFL-NFL-USFL coaching victories (7th on the all-time list)
One of only 14 coaches to win 100 games with one NFL team
The only coach to compete in four Super Bowls in a row
Retired at the age of 72; tied with George Halas as the oldest head coach in NFL history.
First USFL alumnus to be inducted into the Pro Football Hall of Fame
One of only two coaches to appear in both a Grey Cup Championship Game and the Super Bowl. The other is Bud Grant.
Oldest coach ever to win 12 games (age 68) and 10 games (age 71)