According to tradition, the monastery, dedicated to the Virgin Mary, was founded in 1182 by Margrave Heinrich von Ursin-Ronsberg, to house a community that had grown up around a local hermit.

The monastery was first established at the long-abandoned Burg Ursin, the margrave's ancestral castle, where St. Stephen Church's cemetery is now located.

It was saved only by the intervention in 1373 of Anna von Ellerbach, the second founder, sister of the Bishop of Augsburg, and her appointee, abbot Conrad III, known for his extreme frugality.

Irsee recuperated quickly but in 1662 the powerful prince-abbot of Kempten purchased the right of advocacy (German: Vogteirechte), which limited the autonomy of the abbey.

In 1694, following the election of the energetic Romanus Köpfle as the new abbot in 1692, Irsee succeeded in obtaining the status of an Imperial abbey, which it will keep until it was dissolved in 1802.

[5] The construction project was greatly expanded by its successor, abbot Willibald Grindl and the monastery buildings were completely rebuilt.

[7] The abbey celebrated the 600th anniversary of its foundation in 1782 but its prosperity soon came to a brutal end in the aftermath of the French Revolution and Napoleonic campaigns.