William the Conqueror granted a deed to Humphrey de Hoggell (Ogle) to enjoy "all the liberties and royalties of his manor" after the conquest. The Ogle family held the estate from before the Norman Conquest until 1597 when it passed by marriage to the Cavendish family and later to Hollis. Sir Robert Ogle was granted a licence to crenellate in 1341. David II of Scotland was brought here having been captured at the Battle of Neville's Cross in 1346.
Today only the west wing remains from that period. This was the tower house of the medieval tower which had a projecting latrine. Still showing on the western and northern sides are parts of a double moat around a platform 45M across. The manor building that makes up most of today's still standing Ogle Castle appears to be 16th and 17th century work that the tower house was later incorporated into.
- Heritage Gateway, architectural description of Ogle Castle
- Burke, Sir Bernard Burke; John Bernard Burke (1863). A Genealogical and Heraldic Dictionary of the Landed Gentry of Great Britain and Ireland, Fourth Edition, Part II. London: Harrison, Pall Mall. p. 1108.