In 1584 he was captured, judged and sentenced to death by the local Catholics for slandering a priest.

They remained in the castle for almost a century, until 1682 when they moved into the newly completed St. Mary of the Angels.

When the Franciscans moved out, the castle was sold to Antoni Speck who owned it until his death in 1708.

It then passed to Johann Baptist Fortunat and then into the possession of the Sutter family.

On 15 February 1875 Doctor Anton Alfred Sutter became the sole owner of the castle.