Pattinson made his Test debut in the 2011/12 home series against New Zealand, in the first Test in Brisbane on 1 December 2011. He took 5 wickets for 27 runs in the second innings of his debut Test and helped Australia to a nine-wicket win, then took five wickets in the first innings of the second test, and was named Man of the Series, with fourteen wickets at an average of 14.00 across the two tests. He won another Man of the Match award in his next Test, the 2011 Boxing Day Test against India, with match figures of 6/108 and 55 runs. James Pattinson has not been included in the Australia's preliminary for T20 World Cup.
Before the 3rd Test against India in March 2013, Australia dropped Pattinson, along with Shane Watson, Mitchell Johnson and Usman Khawaja following a breach of discipline. Michael Clarke, the captain, revealed that the extreme step had been taken as a result of repeated infractions which led to Watson flying back home and contemplating Test retirement. Former players reacted with astonishment at the harsh decision taken by the team management.
Pattinson's older brother is Darren Pattinson, who played for England in one Test match. Although born in England, Darren Pattinson grew up in Melbourne's outer suburbs. He made his first-class debut for the Victorian Bushrangers and consequently won a contract with Nottinghamshire. Pattinson's girlfriend is Kayla Dickson.  They went to school together and have been together since 2012. Kayla Dickson is also Tory Dickson's sister.