Subsequently his party friends forced Theodore P. Gilman to resign the office of First Deputy Comptroller, and had Kelsey appointed to the post.

On May 2, 1906, Kelsey was appointed by Governor Frank W. Higgins to a three-year term as Superintendent of Insurance, and resigned the comptrollership.

Early in 1907, Governor Charles Evans Hughes asked Kelsey to resign, but he refused.

Then Governor Hughes appointed Matthew C. Fleming a Special Commissioner to examine the Insurance Department, and on February 2, 1908, Fleming declared Kelsey "unfit for the office" in his report to the State Senate, but Kelsey was maintained in office by an even larger majority.

He died on August 20, 1934, in Perry, Wyoming County, New York, after complications from a fall, and was buried in Geneseo.