The first mention of Audley is in the Domesday book of 1086, when it was called Aldidelege (Aldithley), when the lands were held by an English thegn called Gamel. The descent of the Audley and Stanley families can be traced back as far as an Englishman named Ligulf of Aldithley, who held the estate not long after the Domesday survey but whose relationship, if any, to Gamel is unknown. The family later fabricated a Norman origin, at that time more prestigious than an English one, by presenting Ligulf, despite his non-Norman name and the English etymology of Aldithley, as the lord of a fictitious 'Aldithley in Normandy', and his son Adam as a follower of William the Conqueror.
The extended Audley family, originally of Audley Castle but who later built (or re-built) Heighley Castle, Madeley, Staffordshire in 1226, had several additional households including Red Castle at Hawkstone in Shropshire, Buglawton Manor in Congleton, Newhall Tower at Newhall, Cheshire  and a home in Nantwich. In the early 12th century Adam's grandson William married Joan de Stanley heiress of Stoneleigh, Staffordshire thereby becoming William de Stanley of Stoneleigh. This branch of the Audley thus took the name Stanley and Thomas Stanley, 1st Earl of Derby was a direct descendant.
The main line of the Audley family, who had become Baron Audley in 1313, failed in 1391 when Nicholas Audley, 3rd Baron Audley died without a male heir. His sister Joanne married Sir John Tuchet (b 1327) for whose descendant John Tuchet (b 1371) the title of 4th Baron Audley was granted in 1408. He was succeeded by his son James Touchet, 5th Baron Audley. Later generations of the Tuchet's became Earl of Castlehaven.
The Audley and Stanley families and their descendants remained prominent throughout the history of England and Ireland into modern times. The Stanleys were, at one time, one of the richest landowning families in England.
Latham family and "Eagle and child" Coat of Arms
In 1385, Isabella de Lathom, the eldest daughter of parents who had no male heir, married Sir John Stanley and the Latham family (originally Lathom or Lothe Hom - "Brick house" from Lathom, Lancashire) and Stanley families became linked. Following this marriage, the Stanley family adopted the 'eagle and child' crest of the Lathom family and continue to use it to this day.
Several successive generations of the Stanley Earls, along with other members of the family, have been prominent members of the Conservative Party, and at least one historian has suggested that this family rivals the Cecils (Marquesses of Salisbury) as the single most important family in the party's history.
- Baron Audley
- Henry de Aldithley/Audley c. 1175–1246, Lord of the Welsh Marches, governor of Carmarthen castle and Cardigan Castle, Sheriff of Salop and Staffordshire from 1216 until 1221, constable of Shrewsbury Castle and Bridgnorth Castle,Governor of Shrewsbury, Chester Castle and Beeston Castle, governor of Newcastle-under-Lyne. He married Bertred Mainwaring.
- James de Aldithley c. 1220–1272 who married Ela Longespee, daughter of William II Longespee, son of William Longespée, 3rd Earl of Salisbury, a son of King Henry II of England by Ida de Tosny (who would later marry Roger Bigod, 2nd Earl of Norfolk).
- Emma de Audley c. 1224, daughter of Henry de Audley & Bertrade de Mainwaring and great granddaughter of Hugh de Kevelioc, 3rd Earl of Chester, who married Gruffydd II ap Madog, Lord of Dinas Bran Prince of Powys
- Hugh de Audley, 1st Earl of Gloucester c. 1291–10 November 1347
- James Audley founder of the order of the garter c. 1316–1386
- Margaret de Audley, daughter of Margaret de Clare and great-granddaughter of King Edward I of England c. 1318–1347.
- James Touchet, 5th Baron Audley
- Thomas Audley, 1st Baron Audley of Walden Lord Chancellor
- Earl of Derby, first creation and later creations
- Baron Stanley
- Baron Stanley of Alderley
- John I Stanley of the Isle of Man 1350–1414, Lord Lieutenant of Ireland 1413-9, Lord and titular King of Mann from 1406.
- Thomas Stanley, 1st Baron Stanley 1405–1459 King of Mann, Lord Lieutenant of Ireland, Constable of Chester Castle, Comptroller of the Royal Household in 1439.
- Thomas Stanley, 1st Earl of Derby 1435–1504
- Thomas Stanley, 2nd Earl of Derby 1477–1521
- Edward Stanley, 3rd Earl of Derby 1509–1572, Lord High Steward (whose daughter, Jane, was the great great grandmother of Abraham Darby I, founder of the industrial revolution).
- Henry Stanley, 4th Earl of Derby 1531–1593, Lord of Mann
- Ferdinando, the fifth Earl of Derby, Baron Strange of Knockin, 1559–1594, patron of, amongst others, William Shakespeare
- Lady Frances Stanley 1583-1636, daughter of Ferdinando (who married John Egerton, 1st Earl of Bridgewater, son of Thomas Egerton, 1st Viscount Brackley, Chancellor of England, created 1st Lord Ellesmere)
- Edward Stanley, Earls of Derby
- Stanley (surname) List of people with Stanley surname