Except for the portion in Gillsville, the entire highway travels through rural areas of Hall and Banks counties.

SR 323 starts at an intersection with US 129/SR 11 (Athens Highway) southeast of Gainesville in Hall County.

It travels to the northeast to an intersection with SR 82 (Holly Springs Road) and a crossing over the North Oconee River.

The two routes share a brief concurrency over Candler Creek into town, where they pass Pine Grove Cemetery.

Approximately 200 feet (61 m) later, SR 323 splits off to the northeast into Banks County proper, where it crosses over Grove and Hickory Level creeks before it reaches its eastern terminus, an intersection with SR 51 northwest of Homer.