He was a descendant of Jacobite members of the Pilkington Family of Lancashire, England, who fled to Ireland after the Battle of Preston in 1716.

[1] Growing up in a tenement in East Harlem, Pilkington was advised by a doctor to pursue exercise to improve his health and took up boxing at the Union Settlement Association.

[2] Pilkington learned to box by watching Benny Leonard, Harlem Tommy Murphy, and Irish Patsy Cline at his neighborhood gym and based much of his early technique off of what he observed.

During this time, Pilkington lost only 4 out of 150 matches; winning the New York State and Metropolitan Crown boxing titles in the featherweight division.

As a mentor and sparring partner, he is largely credited with helping mold the career of famed boxer Kid Kaplan.