Allman was born in Austin, Texas, brother of David Allman, and the son of Idanell (née Brown; born January 14, 1950; Austin, Texas) and James Martin Allman, Jr. (born April 27, 1950; Travis County, Texas). As he grew up he was talented in soccer and art. Allman played club and school soccer and is deeply interested in contemporary music. After graduating from Austin High School in 2002, he chose to pursue an acting career in Los Angeles over studying art in New York City. Allman is a Christian, and in an interview he stated, "Any time someone tries to represent their idea of God it’s paramount to remember that you’re an imperfect person leading a flawed life."
Allman married American actress Jamie Anne Allman, née Brown of Parsons, Kansas (best known for her role as Terry in The Killing), on June 17, 2006, in Austin, Texas. They have twins (Asher James and Oliver Charles) who were born on January 31, 2013, and a daughter (June Joanne) who was born on May 21, 2014.