Sargent is married to Diane Holloway, former television critic for the Austin American-Statesman. Daughter, Elizabeth, son, Sam.
In a profile published in the January 2005 issue of The Good Life magazine, Sargent stated, "As a newspaper journalist, you're professionally obligated to be fair, accurate, complete and balanced. But there are two pages in the back of the paper where we're obligated to be fair, accurate and complete—but we don't have to be 'balanced.' I'm not a pollster. To me, you're obligated as an opinion journalist to express your views no matter what the politics of the day. If you don't, then people will say, 'Why should I read what this guy's saying? He doesn't even know where he stands?' That makes your position as an opinion journalist kind of useless."