He was a brother of Gov Robert Williams (1766-1836). On 26 Oct 1798 in Caswell County, North Carolina, he married Agnes Payne (1775-1850), a first cousin of Dolley Madison, and they raised 9 children. He was first cousin of the brothers: Robert Williams (1773-1821), John Williams (Tennessee) (1778-1837) and Lewis Williams (1786-1842). He was the grandfather of Lafayette Guild (1825-1870). He died in Tuscaloosa in 1850 and is buried in Greenwood Cemetery. His Tuscaloosa home has been completely renovated.