In 1857 he gained control, but several years later, when Sargent attempted to purchase property needed for expansion, he was met with opposition.
Sargent's ideas on manufacturing were highly progressive and the new factory was so modern with running water on each floor for washing, manufacturing and fire fighting purposes, adequate bathroom facilities and a slate roof, that many in New Haven predicted the failure of the business before it opened.
Many of the early workers of Sargent's factory were Italian immigrants that settled in the Wooster Square neighborhood.
Starting with the production of approximately 1,000 items, by 1871 the company was adding extensive additions to the plant to accommodate 2,000 workmen.
Sargent was the Mayor of New Haven, Connecticut for three terms, serving from 1891 to 1894 and ran an unsuccessful campaign for Governor on the Democratic ticket.