Castle Rock is located 7 miles (11 km) north of Northfield. Nearby places also include Farmington and Randolph. The community is named for a local rock formation. The rock formation is located at the point where 275th ST and Cambodia AVE join, on private property. The rock formation, known to Dakota Native Americans as Inyan Bosdata or "Rock Standing on End" was also the namesake of the nearby Inyan Bosdata Wakpa, known today as the Cannon River. State Highway 3 (MN 3) is nearby.