Pottersville was first called Lamington and afterwards Potters Mills. There were mills here as early as 1756 built and owned by William Willet. One Mill still stands on the left side of County Route 512 heading towards Califon. It was originally used for weaving woolen goods and later turned into a grist mill. The first grist mill was built along the Black River but no longer stands. A commemorative plaque has taken its place.
William Willet owned a day book in which he recorded sales to the Continental Army during the Revolutionary War. His main consideration became supplying the Continental Army. He was paid in Continental currency which around 1780 became worthless. He was ruined financially and was forced to sell both mills to Serrin Potter in 1783, which led to the community's name.
In 1887 upwards of 200,000 baskets of peaches were shipped from Pottersville and New Germantown (Oldwick) by wagons to Chester, Whitehouse and other area communities. The profitable peach growing industry led the Rockaway Valley Railroad to build a spur to Pottersville in 1888.
Black River Falls in Pottersville prompted the railroad to run excursions to the falls. The land around the glen were made into picnic grounds and an amusement park. There was a merry-go-round, dance pavilion and refreshment stand. Some visitors came from Jersey City, N.J. and usually stayed at the Pottersville Hotel. Failure of the peach crop eventually resulted in the end of the Rockaway Railroad. One town resident remembers the park open as late as 1920.