After being drafted by the Sydney Swans at Pick #28 in the 1998 AFL Draft, under the Father-son rule from the Port Adelaide Magpies in the South Australian National Football League, James made his AFL debut in Round 19 of the 1999 season. He suffered numerous hamstring and knee injuries during his career, managing to play just 18 of a possible 162 games (including finals) (11.11% of club games) in 7 seasons with the Swans (1999–2005). He was delisted and re-drafted twice more during his career, in the 2003 Rookie draft and the 2004 National Draft. During his injury-interrupted tenure in Sydney, James built up a reputation as a resilient player and person. He has since taken up a game day role with the club. His father Max James also played for Port Adelaide and South Melbourne, prior to their relocation to Sydney.