Castle Rock is located 7 miles (11 km) north of Northfield. Nearby places also include Farmington and Randolph. State Highway 3 (MN 3) is nearby. The community is named for a local rock formation. The rock formation is located at the point where 275th Street and Cambodia Avenue join, on private property. The rock formation, known to Dakota Native Americans as Inyan Bosdata or "Rock Standing on End" was also the namesake of the nearby Inyan Bosdata Wakpa, known today as the Cannon River.