[2] He is remembered in Sweden for a Pentecost hymn, of which he was the author, which is still popularly sung in churches.

In 1896 he became parish priest of the town of Gällivare in Lapland, then part of the Diocese of Härnösand.

As Bishop of Luleå Bergqvist was responsible for the construction of new churches, and the development of an extended parochial and deanery system.

[2] Bergqvist was politically active, and served as an elected County Councillor for more than twenty-five years, representing the Swedish National Party.

Bergqvist worked extensively with the indigenous Sami population of his diocese, and was a major contributor to the Church of Sweden's project to translate the New Testament scriptures into the Sami language.