He attended the University of Maryland, where he played college football as a quarterback and punter for the Maryland Terrapins. Prior to the 1995 season, Milanovich and four other Maryland players received suspensions for betting on college football and basketball games. Milanovich received an eight-game suspension (later reduced to four) during his senior year for having bet between $25 and $50 on a total of six games. The bets had no impact on the outcome of the games. Despite throwing for 24 TD as a Sophomore and 20 TD as a Junior, Scott struggled mightily as a Senior throwing just 2 TD against 7 interceptions.
Milanovich was named the MVP for the Blue squad in the Blue-Gray Classic, and completed 9 of 20 pass attempts for 175 yards and two touchdowns. As of 2008, Milanovich still held the Maryland record for highest career pass completion rate.
After going undrafted in the 1996 NFL Draft, Milanovich signed as a free agent with the Tampa Bay Buccaneers. During his rookie campaign, he was designated as the team's third quarterback for 15 games, seeing action in one contest. In that game he completed two of three passes for nine yards. In 1997, he was declared inactive before all 16 regular season games and both playoff contests.
Milanovich was expected to be the starting quarterback for the XFL's Los Angeles Xtreme but lost the job to Tommy Maddox. Milanovich saw limited action as the Xtreme's second-string quarterback, behind Maddox but ahead of Noel Prefontaine, whom Milanovich would later coach in Toronto.
Milanovich returned to the CFL in February 2007 when he was named quarterbacks coach of the Montreal Alouettes. A year later he was promoted to offensive coordinator. In 2009 he added the title of assistant head coach.
On December 1, 2011, Milanovich was named the 42nd head coach of the Toronto Argonauts. His Argonauts won the Grey Cup in his first season at the helm. Following a 9-9 regular season in 2012, Milanovich led the Toronto Argonauts to a 35-22 Grey Cup victory in his debut season as a head coach. After winning the 100th Grey Cup in his first season as a head coach in the CFL Milanovich was named the 2012 CFL Coach of the Year.
Milanovich's second season as Argonauts head coach was a successful one. The team battled plenty of injuries yet still managed to finish first place in the Eastern Division with an 11-7 record, their first division championship since the 2007 season. The Argonauts would eventually lose in the Eastern Final playoff game to Hamilton.
Following his time in Tampa Bay he was charged with Driving Under the Influence (DUI) for an incident on April 11, 1998. He was pulled over by police in Pinellas Park, Florida and found to have a blood alcohol level of 0.135. He pled no contest and received a $550 fine, probation, 50 hours of community service and six-month driver's licene suspension. He was able to pay the Salvation Army in order to avoid the community service requirement.