Plumpton Place looks onto the nearby north-facing escarpment of the South Downs, with Plumpton College (formerly Plumpton Agricultural College) and the 11th-century church of St. Michael's and All Angels immediately adjacent to the west and Plumpton village some 500m to the east. There is an entrance formed of two cottages designed by Sir Edwin Lutyens, with a Palladian porch and this leads to his modern bridge over the moat.
Various building materials have been used in the construction of the house. It is believed that the north wing is the earliest, as there is a date-stone of 1568 with the initials I.M. The west wing seems to date from a later period, circa 1600. Over a hundred years later there was a period of rebuilding in brick and some additions by Lutyens. There are lakeside gardens by Lutyens and Jekyll, within large grounds, that include both woodland and pasture.