The first masonry castle was constructed by Vytautas, Grand Duke of Lithuania, in 1412-1413 on an island, now a peninsula, in Lake Asveja in order to secure the capital Vilnius from attacks from Livonia.

After the death of Jerzy, his son Mikolaj "the Red" inherited the property, causing the town nearby to become an important hub for the Reformation in Lithuania.

Barbara Radziwiłł (Barbora Radvilaitė) spent five months in the castle after her marriage to Sigismund Augustus in 1547.

During the Polish–Lithuanian-Swedish wars, the castle was pillaged by armies loyal to the King of Poland and Grand Duke of Lithuania.

The idea of the restoration is actively supported by the representative of the Radvila family, Duke Motiejus Radvila (Maciej Radziwill), who is among the founders of the fund, along with Prof. Albinas Kuncevičius, Dr. Vydas Dolinskas, who was in charge of the restoration of the Palace of the Grand Dukes, and Molėtai District Municipality.