By the time he initially retired in 1888, he held the record for most games umpired in the major leagues (587); he returned to work the last two months of the 1897 season.

[1] The early years of the major leagues were particularly hard on umpires, with hostile crowds and vicious arguments, in addition to the difficulty of constantly working games single-handedly.

Few officials lasted more than two or three seasons, and by the end of the 1885 campaign Kelly had passed Billy McLean's total of 317 major league games as an umpire.

[4] Kelly became a championship boxing referee, officiating Jim Corbett's only successful defense of his heavyweight title against Charley Mitchell on January 25, 1894, in Jacksonville, Florida.

Mitchell's corner called for a foul, but the presence of numerous spectators brandishing guns and eager to defend their bets led Kelly to dismiss the complaint; he later confided that he had been lucky to leave Jacksonville alive.

[5] On July 3, 1899, in Buffalo, New York, Kelly refereed the bout in which Frank Erne won a 20-round decision to take the world lightweight title from Kid Lavigne.