On 21 July 1713, he married Lady Anne Vaughan, a daughter of the 3rd Earl of Carbery. The marriage was not a happy one, and there were no children. In 1728, he began a long-standing affair with the English actress, Lavinia Fenton. Lady Anne died in 1751 and the Duke married Lavinia Fenton on 20 October 1751 at Aachen. She had already born him three illegitimate sons: Charles, Percy, and Horatio Armand Powlett.