The accounts of his early life vary: the Australian Dictionary of Biography states that his father was a bank manager, while other sources suggest that he was raised in low socio-economic conditions and relied on benefactors to support his career.

Charlton was a member of North Steyne Surf Life Saving club in his teens, before transferring to Manly Life Saving Club in the mid 1920s[2] Charlton first came to public attention in 1921 when he won a 440 yd freestyle race in the open division at a New South Wales Swimming Association competition in 5 min 45 s. It was his youth that led to his nickname "Boy".

In 1922 Bill Harris, the bronze medallist in the 100m freestyle at the 1920 Summer Olympics, came to Australia from Honolulu to compete against the likes of Frank Beaurepaire and Moss Christie.

Charlton defeated Harris at the New South Wales Championships, winning the 440 yd in 5 min 22.4 s. He then set a world record of 11m 5.4s in the 880yd event, as well as winning the one mile race in 23 min 43.2 s. Charlton used a trudgen stroke which embodied characteristics of the modern crawl stroke, which was at the time in its infancy.

The start of 1924 in Australia was highlighted by the arrival of Swedish swimmer Arne Borg, at the time the holder of four world records, to compete against the 16-year-old Charlton in the 440yd freestyle at the New South Wales Championships.

[4] They again met in the 880yd and 220yd events, with Charlton winning the former in a world record time of 10 min 51.8 s and the latter in an Australian record of 2 min 23.8 s. Charlton was selected for the Australian team for the 1924 Summer Olympics and travelled to Paris by sea with his coach, Tom Adriann, who was also appointed the team coach.

Charlton forged ahead to lead by 5 metres at the 600 m, before proceeding to defeat Borg by 40 m, while lapping the remainder of the field to win gold in a new world record time of 20m 6.6s.

Charlton then combined with Ernest Henry, Moss Christie and Beaurepaire to claim silver in the 4 × 200 m freestyle relay behind the United States.

Physiologists had become involved in sport at the time of the Paris Olympics and Charlton's lung capacity was tested with a machine, which blew mercury through a set of bent tubes.

After the games, Charlton declared that swimming would take a back seat to his study and work career, and declined offers to tour the US and Europe.

He resumed studies at Hawkesbury Agricultural College, but did not graduate and subsequently became a station-hand at Kurrumbede station in Gunnedah, in western New South Wales.

Charlton limited his training to irregular visits to Sydney, when he consulted his coach, former Olympic medallist Henry Hay.