Instead, he engaged in mercantile pursuits in New York with Ely, Vanderpoel & Kitchell which was founded in 1868.

He was not a candidate for renomination in 1872, however he was again elected as a Representative to the Forty-fourth Congress serving from March 4, 1875, until the date of his resignation on December 11, 1876.

While in office, he gave a speech on July 29, 1876, regarding the method of manufacturing the circulating notes and securities of the Government.

Ely, who never married, died at the age of 86 on July 1, 1911, in Livingston, Essex County, New Jersey.

[2] His estate, valued in excess of $1,000,000, was almost entirely left to charity, including $50,000 to a fund for juveniles to "help offending youths get a better start in the world," $100,000 to the American Sunday School, $100,000 to United Charities, as well as funds to Orange and Morristown Memorial Hospitals.