[1] At the 1908 London Olympics, Baker represented Australasia in swimming and diving, as well as taking part in the middleweight boxing event, in which he won a silver medal.

[3] However, "His stature as an athlete depends largely upon the enormous range rather than the outstanding excellence of his activities; it was as an entrepreneur-showman, publicist and businessman that he seems in retrospect to have been most important.

"[3] Baker retired from competition after being injured in a motor-vehicle accident, and became involved in boxing promotion, bringing a number of top fighters from North America and Europe to fight in Australia.

During this time, he began to act in a number of silent film roles which showcased his horsemanship, including The Enemy Within, The Man from Kangaroo, and The Shadow of Lightning Ridge.

He appeared at the 1908 London Olympics, representing Australasia in the 4 × 200 m freestyle relay, finishing fourth, and in the diving where he lost in the first round, coming seventh against the powerful Germans who dominated the sport at the time.

Working with a dubious, larger-than-life character by the name of Hugh D. "Huge Deal" McIntosh, Baker became one of the premier boxing promoters in Australia, bringing to the country many of the best boxers of the day from North America and Europe.