He was born in the Charente department of France, as the second son of Louis Du Pont Duchambon and Jeanne Mius d’Entremont de Pobomcoup, making him Acadian on his mother's side.
Vergor began his military career as a cadet in the Compagnies Franches de la Marine and served in various North American locations, mostly in what would later be the Maritime Provinces of Canada.
An injury suffered in 1745 during the siege of Louisbourg resulted in his being sent to France for several years, along with nearly all the other French people on Cape Breton Island.
During Father Le Loutre's War, de Vergor was engaged in the Battle off Port La Tour (1750) in Nova Scotia.
Two years later, Vergor was a part of the defense of Quebec under Louis-Joseph de Montcalm during the siege prior to the Battle of the Plains of Abraham.