He was a brother of Gov Robert Williams (1766-1836). On 26 Oct 1798 in Caswell County, North Carolina, he married Agnes Payne (1775-1850), a first cousin of Dolley Madison, and they raised 9 children. He was the grandfather of Lafayette Guild (1825-1870). He died in Tuscaloosa in 1850 and is buried in Greenwood Cemetery. His Tuscaloosa home has been completely renovated.