Unterkirchberg (Lower Kirchberg) had a Roman castrum built around 40 AD as part of the Upper Germanic-Rhaetian Limes to secure the street along the south shore of the Danube.

It was finally sold by Emperor Maximilian I. to Jakob Fugger in 1507, together with the adjacent lordships Weißenhorn, Wullenstetten and Pfaffenhofen (Roth), for 50.000 guilders.

The emperor granted him the title of Imperial Count in 1514 so the former burgher could operate his business without interference from local nobility.

[3] A new castle was built by the Fugger family in 1767; its architect was Franz Anton Bagnato.

Today it is owned by countess Maria-Elisabeth von Thun und Hohenstein, née countess Fugger zu Kirchberg und Weißenhorn.