Brown was born in Colchester, Essex, England, and went to Colchester Royal Grammar School, which he left at fifteen. His first job working in a factory was the stimulus for Smallcreep's Day. In 1960, Brown was one of a small group who sat down and blocked the entrance of the Atomic Energy Research Establishment at Foulness. He was sentenced to six months in jail when he refused to agree not to repeat the action. In the following year he was one of the "Eskimo Navy" which boarded Polaris submarines in kayaks, resisting the establishment of the base at Holy Loch. He was a member of the Committee of 100 founded by Bertrand Russell to organise mass nonviolent resistance to nuclear war. He married in 1962, and his first child was born later that year. The following year he moved to a small cottage in rural Gloucestershire, where he raised his 4 children. He worked at various jobs locally, including Dursley's main factory, Lister's and Peter Scott’s Wildfowl Trust in Slimbridge. In 1966 he set up a craft pottery he called The Snake Pottery. He later gave up employed work to concentrate on the pottery. In 1987 he separated from his wife and moved out of the cottage.