Oxford United signed Ruffels on a free transfer on 30 July 2013. He made his debut as a substitute in a 3–1 league win over Torquay United on 17 August 2013 and his first starting appearance in a 2–1 home defeat to Portsmouth in the Football League Trophy on 8 October 2013. He scored his first senior goal in a 1–1 League Two draw versus Bury on 4 February 2014. On the 17th April 2014, Josh signed a new contract keeping him at the club for a further 2 years.  9 days later, he was voted 'Young Player of the Year' after he established himself in the first team in his first full season for the club.