Joseph Dupont Duvivier was born in Port Royal, Acadia, the second of the three surviving sons of François du Pont Duvivier and Marie Mius d'Entremont de Pobomcoup.
Like his brothers, he entered the military service at Île Royale, but didn't take part in their business endeavours.
He fought in King George's War and distinguished himself at the first Battle at Port-la-Joye (1745).
During the French and Indian War he was present at the Siege of Louisbourg (1758) and sent back to France after the capitulation.
But this expedition was intercepted by the British navy on 3 July and defeated in the Battle of Restigouche.