The first house here was acquired by Peter des Roches, Bishop of Winchester, for his bishopric early in the 13th century. This house was subsequently torn down in the latter half of the 15th century by William Waynflete, Bishop of Winchester and Lord Chancellor, to make way for a large brick residence with a tower gatehouse (Waynflete's Tower, see below) that still stands today. Cardinal Wolsey, who possessed Esher Place as Bishop of Winchester, was kept under house arrest here after his fall from power. The estate was then seized by Henry VIII, restored to the Bishop of Winchester by Mary I, and the lease was then re-purchased by the Crown under Elizabeth I, who granted it to her Lord High Admiral, Lord Howard of Effingham, who granted it to Sir Francis Drake's cousin Richard Drake. After the defeat of the Spanish Armada, three captured Spanish admirals and their retinue were held at Esher for over five years. It remained in the Drake family until 1634. Over the next 75 years, Esher Place was held by at least seven individuals, the last two being Sir Thomas Lynch, an early English governor of Jamaica, and John Latton, a pluralistic place-holder under William III.
In 1716 the estate was separated from the manor house, with the estate being sold to the first Duke of Newcastle, who owned nearby Claremont. The house and grounds, after passing through two more owners, including Peter Delaperte, one of the directors of the South Sea Company, finally came into the ownership of the Duke of Newcastle's younger brother, Henry Pelham, in 1729. Pelham hired William Kent to renovate the property: Kent did so by pulling down much of the medieval and Tudor portions of the building—except for the gatehouse—and adding wings and some of the earliest Gothic revival ornamentation in England. In The Seasons, the Scottish poet James Thomson (author of the lyrics to Rule, Britannia!) mentions "Esher's peaceful grove/Where Kent and Nature vie for Pelham's love". The inhabitants of Esher Place also had the privilege of using one-half of the Newcastle Pew at St. George's Church, Esher. After Pelham's death, the property passed to his daughter, and was then purchased in 1805 by London merchant John Spicer.
Spicer pulled down the entire house and used the material to build another, designed by Edward Lapidge, on a more elevated site. The new house was stuccoed, in imitation of stone, with Ionic porticoes on north and south fronts, and semi-circular wings. In the late 1890s, this building was then incorporated as a wing into the current French Renaissance style house on the site, designed by G.T. Robinson and Achille Duchêne at the behest of Edgar Vincent, later 1st Lord D'Abernon. Vincent had purchased the much-reduced estate in 1895 from Money Wigram[disambiguation needed], who had bought it from the Spicer heirs. Vincent—Lord D'Abernon after 1914—had many famed guests, including Edward VII when Prince of Wales, Cecil Rhodes, and Anna Pavlova. However, in 1930 Lord D'Abernon gave the house to the Ragged School Union, later the Shaftesbury Society, while most of the grounds were sold to developers who built a housing estate around the mansion. The house became the Shaftesbury Home for young children from 1930 until 1952, when it was sold to the Electrical Trades Union (now merged into Unite the Union), which opened it in 1953 as the college it remains today.
Waynflete's Tower ( gatehouse, originally built at the same time as the late 15th-century house, but much modified by William Kent following Henry Pelham's purchase of the property. Kent added the two three-bay ranges of three stories to the left and right, as well as the one-story porch between the turrets with ogee-arched doorway, triple window above with ogee-headed lights, and quatrefoil windows. However, the rib-vaulted entrance hall was the gateway in Wayneflete's time, though stuccoed over by Kent. The Tower is currently a residence.) is a grade-1 listed
- English Heritage. "Details from listed building database (286725)". Images of England.
- Nairn (2002), p. 222-23.
- Cust, Lionel Henry (1892). "Lapidge, Edward". In Lee, Sidney. Dictionary of National Biography 32. London: Smith, Elder & Co.
- Prosser, G.F. (1828). Select Illustrations of the County of Surrey. London. Retrieved 24 July 2011.
- Website with historical information
- English Heritage. "Details from listed building database (286914)". Images of England.
- Website for the tower
- Website discussing Time Team investigation
- Esher Place Training and Conference Center: A History
- Nairn, Ian; Pevsner, Nicholas (2002). Cherry, Bridget, ed. The Buildings of England: Surrey (Revised 2nd ed.). Yale University Press. ISBN 0-300-09675-5.
- English Heritage's PastScape website entry