The cornerstone for the stone church was laid on June 9, 1864[4] by John Henni, first Bishop of Milwaukee.

Soon thereafter, Oschwald commissioned work on the Loreto Shrine Chapel, dedicated to Our Lady of Loretto, completed in 1870 and expanded in 1872.

[4] Dissension arose in the community soon thereafter, and legal battles took their toll on membership and finances of the Association.

As the original generation of settlers died out, the future of the village, and its parish, fell into doubt.

Louis Barth, who had lived in St. Nazianz and studied at the Oschwald Seminary, learned of the Society of the Divine Savior (the Salvatorians) and invited its founder, Fr.

The church was extensively remodeled in 1926, but the Great Depression put a halt to further improvements to the properties for some time.

In 2000, the diocese undertook a massive reorganization intended to address demographic changes and the priest shortage.

In 1982, the entire complex of St. Gregory's Church, the Loretto Shrine Chapel, and St. Mary's Convent was added to the National Register of Historic Places as .