Horace Seymour was the son of Admiral Lord Hugh Seymour (son of Francis Seymour-Conway, 1st Marquess of Hertford) and Lady Anne Horatia Waldegrave. He married, firstly, Elizabeth Malet Palk, daughter of Sir Lawrence Palk, Bt., on 15 May 1818. He married, secondly, Frances Selina Isabella Poyntz, daughter of William Stephen Poyntz and Hon. Elizabeth Mary Browne, in July 1835.
By his first wife he had three children;
- Lt.-Col. Charles Francis Seymour (13 Sep 1819 - 5 Nov 1854), killed at the Battle of Inkerman
- Frederick Beauchamp Paget Seymour, 1st Baron Alcester (12 Apr 1821 - 30 Mar 1895)
- Adelaide Horatia Elizabeth Seymour (27 Jan 1825 - 29 Oct 1877), who married Frederick Spencer, 4th Earl Spencer, and was an ancestor of Diana, Princess of Wales.