When Samogitia was handed over to the Order, the Teutons built a small castle of Koenigsburg on this spot, housing 80 knights and 400 soldiers.
Since then it was the personal property of kings of Poland and the Grand Dukes of Lithuania until 1549, when Sigismund II Augustus donated it (along with the surrounding town) to his wife, queen-consort Barbara Radziwiłł.
Between 1653 and 1664 Prince Janusz Radziwiłł ordered its reconstruction and refurbishment, which gave it its current form.
The manor was known to house that family's extensive art collection including works by Leonardo da Vinci, Rubens, Caravaggio and Jan Matejko.
Currently it houses the Lithuanian Institute of Melioration (Lietuvos žemes úkio inžinierijos institutas), as well as a small museum devoted to both the Tyszkiewicz family and Lithuanian composer Juozas Naujalis born in the nearby village.