Richards studied at the local state school for Aboriginal children and worked cutting timber with his father at age 14.
His earnings allowed him to buy four houses He married Dorothy Elizabeth Iselin in St Luke's Church of England, Brisbane, on 14 December 1935.
He eventually used up all his earnings and the native affairs branch of the Queensland government was requested to take jurisdiction over him after police in Sydney charged him with vagrancy in May 1947.
While in Sydney he was arrested for 'vagrancy and drunkenness' and, under the 'Queensland Act', was sent to the Aboriginal reserve on Palm Island, Queensland, where he tended gardens and refused to talk about his boxing career.
At some point, he returned to Sydney after his daughter sent word of his estranged wife falling ill.[3] He died of a heart condition in Dulwich Hill in 1967.