Born in Baltimore, Maryland, he was the younger brother, by two years, of Thomas Joseph Kelly.
In the mid-1930s, the brothers went to work for J. W. Y. Martin, a trainer and racehorse stable proprietor who owned Snow Hill Farm, in Glyndon, Maryland in the Worthington Valley.
However, both brothers had their careers interrupted when they joined the United States armed forces during World War II.
In 1949 he was hired by Baltimore clothing manufacturer Harry Isaacs to train for his Brookfield Farm racing stable.
He also owned horses he trained and raced including Lucky Coin, winner of the Kelso Handicap who also set a Belmont Park turf course record of 1:32.00 for one mile.