Born in Preston, Penny stayed on at school after the age of fourteen as a pupil-teacher.
That year, he was a founder of a local branch of the Fabian Society, then in 1893 was instrumental in it becoming part of the new Independent Labour Party (ILP).
[1] Penny spent much of his time on the ILP, and in 1895 became its assistant general secretary, serving under Tom Mann.
In 1898, Mann resigned and Penny replaced him as general secretary, moving to London.
He was a founder of the Planet Friendly Society in 1905, and by 1908 had moved to Sheffield to work as the organiser of its local district.