Victor de Broglie was born in Rome, Italy where his father, monarchist politician Albert, 4th duc de Broglie, held a diplomatic post.
On 26 September 1871, he married Pauline de La Forest d'Armaillé (1851–1928) in Paris.
With her, he had four children who survived to adulthood, including two sons, Maurice and Louis, both of whom were physicists, and both of whom would hold the ducal title.
De Broglie acceded to the title of duc de Broglie on his father's death in 1901 but died only a few years later, passing the title to his eldest son, Maurice.
Maurice had no surviving children, while Louis died unmarried, and the title passed collaterally to Victor-François, a descendant of the 5th Duke's third younger brother.