The park was originally wooded and, together with what is now Maryon Wilson Park, was known as Hanging Woods. This was a wild wooded area and formed an ideal retreat for highwaymen who robbed travellers on Shooters Hill and Blackheath. Those who were caught were hanged here – hence the name.
The park includes Cox's Mount, which was used by the Romans as a hill fort, which was discovered in 1915. In the 1850s, Cox's Mount was used to help ships on the nearby River Thames adjust their compasses.