It takes its name from the special appearance of the freestanding 35-metre (115 ft) high sandstone rocks (its name literally means "column"), into which it was carved in the 14th century.

During the 15th century, Knight Mikeš Pancíř of Smojno was based here and launched a raid on Lužice, and as a consequence the castle was besieged in 1445–45.

Historical documents talk about several men: Konstantin was a bricklayer; Vaclav painted a solar clock on the wall; and an optician and gardener called Samuel grew corn on the plains around the castle, as well as wine on its southern terraces.

After the hermitage ceased to exist the state owners, the Kinsky family, turned the complex into a romantic tourist attraction.

Little has been preserved of the original structure on the elevated plateau, although the spaces sunk into the rock can be clearly seen today like the dungeon, well, hermitage, passage and the terraced walkways.