Herbert Sweet was born on October 8, 1919, in Hartford, Connecticut; the following year his family moved to Troy, New York, where he grew up and received his schooling.

There, he earned the Bronze Star Medal with valor device for exposing himself to enemy fire in order to rescue his wounded company commander, two other Marines and a corpsman.

Sweet returned from Korea in July 1952, and served thereafter as assistant to the professor of naval science, NROTC Unit at Columbia University in New York City for two years.

Following his second tour of duty at Camp Lejeune, he was stationed in the Philippine Islands as barracks sergeant major, Subic Bay, from 1958 to 1960.

While serving in this capacity, he was selected as the 4th Sergeant Major of the Marine Corps by a board convened in Washington; he assumed his new post on July 16, 1965.