SR 323 starts at an intersection with US 129/SR 11 (Athens Highway) southeast of Gainesville in Hall County. It travels to the northeast to an intersection with SR 82 (Holly Springs Road) and a crossing over the North Oconee River. Just before entering Gillsville, the route intersects SR 52. The two routes share a brief concurrency over Candler Creek into town, where they pass Pine Grove Cemetery. At the intersection with County Line Road, they begin to travel along the Hall–Banks county line. Approximately 200 feet (61 m) later, SR 323 splits off to the northeast into Banks County proper, where it crosses over Grove and Hickory Level creeks before it reaches its eastern terminus, an intersection with SR 51 northwest of Homer. Here, the roadway continues as John Morris Road.