He studied at St. Joseph Seminary in Covington, Louisiana, and at the Pontifical College Josephinum in Worthington, Ohio.
On November 4, 2004, Herzog was appointed Bishop of Alexandria, Louisiana, by Pope John Paul II.
He received his episcopal consecration on January 5, 2005, from Archbishop Alfred Hughes, with Bishops Thomas Rodi and Robert Muench serving as co-consecrators.
In 2013, Herzog released a statement regarding a retired Alexandria priest named Frederick James Lyons.
Herzog underwent extensive physical and occupation therapy, eventually submitting his resignation in 2016.