His father died when Rogers was 10 years old, and after his mother's remarriage he was raised primarily by an uncle in Lake George.

He attended Granville Academy, and graduated from Union College in 1818, where his classmates included William Henry Seward, Sidney Breese, Alonzo Potter and Augustus Porter.

His father had been a successful lumberman and merchant, and Rogers was financially secure enough to pursue life as a gentleman farmer in Sandy Hill, along with research in botany and other scientific fields and writing on politics and other topics.

During his term, Rogers was chairman of the Committee on Expenditures in the Department of State, and advocated the abolition of slavery in the District of Columbia.

During the American Civil War he supported the Union and worked to recruit soldiers for New York's volunteer regiments.