Silvagni did not play in the TAC Cup as a junior, but instead played for his school, Haileybury College winning a premiership in his final year in 2005. He joined the Casey Scorpions in the Victorian Football League in 2006, winning selection in the VFL's under 23 team. He had two successful seasons in 2008 and 2009, playing mainly at full back, but also pinch-hitting in the ruck or forwardline, finishing 5th in 2008 (2nd best non-AFL aligned player) and then third in 2009 in Casey's best and fairest award, being awarded the most consistent award in 2008 and winning selection in the 2009 VFL Team of the year.
After playing well in Fremantle's three NAB Cup pre-season games, Silvagni was promoted to Fremantle's senior list and was selected to make his AFL debut in the opening round of the 2010 AFL season.
Silvagni made his debut for Fremantle in the opening round of the 2010 AFL season and performed well, taking 7 marks and gathering 14 possessions, including an impressive 55-metre goal as Fremantle recorded a 56-point victory over the Adelaide Football Club. The celebrations, however, were tempered when Silvagni's former captain at the Casey Scorpions, Kyle Matthews, was knocked unconscious when he was involved in a fight after watching the game at a hotel in Hawthorn.
His drafting as a "mature-aged rookie" and subsequent composed performances in his first few games are seen as an indication of a change to the recruiting philosophy to not only draft the best 18-year-old players, but also to consider older players.