In written sources from the early 15th century, Gelgaudiškis is regarded as a part of Skirsnemunė manor.

In 1504 or 1507, the Grand Duke Alexander granted Gelgaudiškis with the woodland from the Nemunas to Siesartis rivers to the nobleman Ivanas Sapiega [lt].

Only after the involvement of King Sigismund the Old was the manor returned to the widow, and after her death, it became the subject of disputes between her stepsons.

Who quickly leased the manor with all its buildings and land, including the Zapyškis estate, to Grand Stolnik of Lithuania, the elder of Kaunas and Telšiai, Jan Hieronimowicz Chodkiewicz.

The manor also included several residential houses, barns, a few granaries, threshing floor, a malthouse, a bakery, and a sauna.

It was bought by a German baron, Theodor Friedrich von Keudell, born in 1751, Friedrich von Keudell bought only the Lower Gelgaudiškis for 75 000 ducats on preferential terms (which were granted to the German baron by Prussia, which had occupied the region).

In 1818, the owner of the manor, Teodor von Keudell, applied to the authorities of the Augustów Voivodeship for permission to establish a new Evangelical Lutheran parish.

Gustav von Keudell, who inherited the estate while still a minor, built a new main manor house between 1842 and 1846.

[1] The von Keudells began to bring Germans to the lands of Gelgaudiškis, attempting to Germanise the region.

During the 1863 Uprising, the manor was attacked by the rebels, who seized the estate's treasury with 1 500 silver roubles.

[4] At the end of the 19th century, a Tsar's decree forbade foreigners to own estates on the border territory of the Russian Empire, so the von Keudells were forced to sell the estate and moved to Dresden[5] In 1898, the manor of Gelgaudiškis was bought by Ona Konča-Komarienė, who did not have time to enjoy it to the full - she died a year and a half later.

When Gelgaudiškis was bought by Komar family [lt], the new owners substantially rebuilt the house without altering the exterior.

In keeping with the old traditions of the manor, there was also a gunroom on the ground floor, where the Medard Komar, who was not fond of hunting himself, kept a large collection of guns and old suits of armor.

They donated the entire mansion to the Archdiocese of Kaunas, which, with the help of the Lithuanian Women's Care Committee [lt], established an orphanage, which operated until 1941.