Grafton is the son of James Oliver Charles FitzRoy, Earl of Euston (1947–2009) and his wife, Lady Claire Amabel Margaret Kerr, the daughter of Peter Kerr, 12th Marquess of Lothian. He was known as Viscount Ipswich from his birth until he succeeded to the dukedom. On his father's death in 2009 he could have taken the title of Earl of Euston, but did not do so.
From 2003 to 2005 he lived in Nashville, Tennessee. He worked for the music business management firm FBMM, was a part-time radio show host on 91.1 WRVU, and played drums with a band called 'The Squibs'. He also plays the guitar. From 2005 to 2007 he worked for the Rolling Stones as merchandising co-ordinator for their "A Bigger Bang Tour".
In 2007 he moved to London and in 2009, due to the death of his father, he returned to Suffolk to help manage the Euston estate. He currently promotes live music events through the Euston Estate.
On 14 August 2010 Lord Ipswich, as he was then known, married Olivia Margaret M. Sladen at Snowshill, Gloucestershire. Their engagement had been announced 15 April 2010. Their son, Alfred James Charles, Earl of Euston, was born on 26 December 2012.