A wooden castle is believed to have stood on the site in mediaeval times.
The stone-built castle known as Lathom House, built by the Stanley family in 1496, had eighteen towers, and was surrounded by a wall six foot thick and a moat eight yards wide, its drawbridge defended by a gateway tower. In the centre of the site was a tall tower known as the Eagle Tower.
Lathom House was the last Royalist stronghold in Lancashire during the English Civil War and was twice besieged by Parliamentarian forces. During the first Siege of Lathom House by Sir Thomas Fairfax in 1644, the house was defended by Charlotte, Countess of Derby and 300 men who kept possession until Royalist forces under Prince Rupert of the Rhine arrived in the area en route to attack Bolton. After the siege the countess and her retinue fled to the Isle of Man. In 1645 the house was again besieged by General Egerton with 4000 Parliamentarian soldiers, and was surrendered after a protracted siege after which the fortifications were demolished by the Parliamentarians. James Stanley, husband of Charlotte was beheaded in Bolton in 1651 for his part in the Bolton Massacre. The Stanley manors were confiscated by Parliament.
After the Restoration of the monarchy in 1660, Lathom House was returned to the Stanleys and remained with them until 1730 when it passed by the marriage of Henrietta Stanley to John Ashburnham, 3rd Baron Ashburnham who sold it.
It was subsequently bought by Sir Thomas Bootle who commissioned Giacomo Leoni to rebuild the house as the finest Palladian house in the county. Built over 15 years from 1725-40, its deer park was designed by renowned landscape gardener Humphry Repton and boasted The Lines which, at two-and-a-half miles long, were the grandest 17th-century tree-lined avenues in England. There was also a temple, a hunting lodge and a Gothic dairy. The north front of the mansion measured 156 feet, and its surrounding parkland was between three and four miles in circumference.
It passed through his niece to Richard Wilbraham Bootle and their son, Edward, Lord Skelmersdale. The latter's grandson Edward inherited and was made Earl of Lathom. Edward's son, the second earl, was accidentally shot and died in 1910. The last resident of Lathom House was Edward Bootle-Wilbraham 3rd Earl of Lathom (and 4th Baron of Skelmersdale). During the First World War the hall was used for military purposes, mainly the training of horses, and after the war the third earl decided not to renovate and reoccupy it but to live instead at nearby Blythe Hall.
The estate was sold in 1920 to a London consortium. After a brief period as a boys’ school, the main block of the house demolished in 1925 and now a 1950s office block stands on the spot.  The estate land of 4000 acres was sold piecemeal, mainly to the tenants. The remaining west wing of the house was converted to apartments.
The ruins of Lathom House were excavated by the Historical Council of Northern Lancashire, in its efforts to reconstruct the 18th-century buildings; the team on site came across medieval foundations and have tried to salvage them. While the main buildings became uninhabitable several decades ago, there are almshouse cottages neighbouring the Lathom Hall Chapel.
- Foxe's Book of Martyrs by John Foxe
- Lewis, Samuel (1848), "Lathom", A Topographical Dictionary of England (British History Online): 30–33, retrieved 2011-09-18
- Farrer, William; Brownbill, J, eds. (1907), "Lathom", A History of the County of Lancaster: Volume 3 (British History Online): 25–34, retrieved 2011-09-18
- England's lost Downtons Daily Mail, 16 February 2012
- Robinson, p. 136
- Robinson, John, Felling the Ancient Oaks, Aurum Press, 2011, ISBN 978-1845136703