Kelly played his junior football with Collingwood and was promoted to their seniors in 1930. Despite playing the Preliminary Final, Kelly was omitted from the Collingwood team which won the 1930 VFL Grand Final. He spent the 1932 season at Beechworth before returning to Collingwood for one final year. Used on the wing or as a centreman, Kelly crossed to Melbourne in 1934 for three seasons and then Essendon for another three.
He captain-coached Port Melbourne to the Victorian Football Association (VFA) premiership in 1940 and remained in that role the following year. In 1942 he returned to the VFL and joined his fourth club, St Kilda, for whom he kicked 21 goals that year and would captain in the 1944 season. Kelly later played at Yallourn North as captain-coach.