He won back his seat in 1911 and became vice-chairman of the Board of Aldermen in 1912, promising to enforce all rules fairly from the chair (including those against smoking).

Despite his stated intention of keeping all the department heads appointed by his predecessor for the rest of his term, Kline, in his very last days of office, dismissed Rhinelander Waldo as Commissioner of Police rather than accept a New Year's Eve resignation.

[5] He later served as a Republican U.S. Representative from New York (5th District in Brooklyn) from 1921 to 1923, being named to the House Committee on Naval Affairs,[6] but lost re-election in 1922 to Loring M. Black, Jr. (Democratic, 1923–1935).

Kline spent all of his post-Congressional life as New York manager of the sea-service bureau of the United States Shipping Board.

Here are the election returns from the Fifth Congressional District in Brooklyn for 1920–1922, as reported by William Tyler Page, the Clerk of the United States House of Representatives.