The castle is located on a hard sandstone slab, 140 metres above the Polenz valley and is the major landmark of the small town.

In 1353 the castle went into the possession of the Bohemian nobleman, Hynek Berka of Dubá, whose coat of arms with crossed oak branches decorates the entranceway to the second courtyard.

In 1443 the Berkas of Dubá lost the estate through exchanges and purchase, only mentioned for the first time under their name, to the Electorate of Saxony under Frederick the Humble, although it remained a Bohemian fief until 1806.

The original wooden structures were gradually replaced during the 17th and 18th centuries by the present stone buildings and even successfully withstood a Swedish siege in 1639.

From 1949 it was extended to become the largest youth hostel in the GDR; at the end of the SED rule an internment camp for 890 political opponents was planned.