The traditional owners of the region are the Jaara people, part of the Djadjawurrung language group of the Kulin nation.

The names of surrounding agricultural districts reflect the legacy of the First Nations inhabitants: Barrakee, Buckrabanyule, Woosang, Wooroonook and Yeungroon.

Unlike many other towns in the region, Charlton was not established as a result of gold mining, but for pastoral purposes and the proximity to a permanent water supply provided by the Avoca River.

A prominent landmark is the former hotel, the "Vale of Avoca", built in 1879 by James and Elizabeth Egan, on the site of an early bark hut inn.

Constructed on a reef of stone, Mrs Egan named the building as being in the "vale" or valley of the nearby Avoca River.

In February 1960, Rena Richardson married William Wood and, in 1961, she sold the Tribune to Ian and Coral Cameron.

Sheep and cattle farming are also prolific with the largest beef feedlot in the state, situated in Yeungroon just south of the town.

Charlton enjoys a temperate Mediterranean style climate, with warm dry summers and cool cloudy winters.

Along the river is a 2 km walking track connecting High Street to Charlton Park, where sporting and recreational facilities exist.