He was appointed in 1872 as the first director of Harvard University's Arnold Arboretum in Boston, Massachusetts, and held the post until his death.

Sargent enlisted in the Union Army later that year, saw service in Louisiana during the American Civil War, and was mustered out in 1865.

Under his direction, the family estate became a landscape without flower beds or geometric arrangements, but rather a recreation of nature with winding lanes, overhanging branches, and a profusion of trees and shrubbery.

He was colder than the surrounding, and notoriously chilly, Boston society; had nothing to do with local government; and cared little for the social ills of his era.

[1] In 1888, he became editor and general manager of the weekly Garden and Forest, "a journal of horticulture, landscape art, and forestry".

These include: In 1913, botanists Alfred Rehder & Ernest Henry Wilson published Sargentodoxa, a genus of flowering plants from China and Indo-China, belonging to the family Lardizabalaceae and was named in Charles Sprague Sargent's honor.

[8] After Sargent's death in 1927, at an Arbor Day memorial ceremony, Massachusetts Governor Fuller planted a white spruce on the grounds of the Massachusetts State House in his memory, and noted: Professor Sargent knew more about trees than any other living person.