This region was first inhabited by the Māori, who settled at a locale known as The Neck, which is a long peninsula that extends across the mouth of the inlet from the southern coast. The only town on Stewart Island, Oban is located on the north coast of the inlet, close to its mouth. In earlier times, the inlet was used as a base for whaling. Southern right whales were the main target in this area and over expoited. Fortunately, however, their sightings around the island are increasing recently.
Paterson Inlet has three main arms; North Arm and South West Arm lie at the upper reaches of the inlet, 15 kilometres from its mouth. Big Glory Bay, the third arm of the inlet, lies behind The Neck in the southeast of the inlet. The inlet drains the Rakeahua and Freshwater Rivers, the latter of which drains a large swampy valley that covers much of the northern part of the island.