St. James is located in St. James Township at the northern end of Beaver Island, the largest island in Lake Michigan. The center of town faces southeast onto St. James Harbor, through which the Beaver Island Ferry arrives from Charlevoix. The northeastern side of the entrance to the harbor is marked by the Beaver Island Harbor Light. The St. James CDP extends east about 0.5 miles (0.8 km) to Lookout Point and Sucker Point, marking the northeastern end of the island.
According to the U.S. Census Bureau, the CDP has a total area of 1.03 square miles (2.67 km2), all of it land.