Earl Cowper, who later became Lord Chancellor of Great Britain, acquired the Cole Green estate c. 1700. He remodelled the estate in 1704 and made alterations to the house in 1711. The 5th Earl Cowper commissioned Samuel Wyatt and then William Atkinson to design a new house in a slightly different location in Regency-Gothic style; construction work started on site in 1806. The park was landscaped with advice from Humphry Repton, starting in 1799. Following the death of the 7th Earl Cowper in 1905, the estate was inherited by Lady Desborough and, after she died in 1952 with no heir, the estate was sold in lots by auction the following year but there was no interest in the house itself and it was demolished between 1953 and 1954.
Panshanger Park is owned by Lafarge who extract sand and gravel from the site. Although Panshanger House was demolished, the orangery, nursery garden wall, stables, and a number of cottages and estate buildings remain, all of which are listed by English Heritage.