They had a palace (Schloss) built in the years 1585–1588 designed by the architect to the Elector, Hans Irmisch.

In 1728 a fire destroyed a large part of the town of Frauenstein and affected both the castle and the palace very badly.

Today the 13th-century tower house, known as Dicker Merten and the 1+1⁄2-metre-thick (4.9 ft) curtain wall remain standing.

The remaining rooms include the cellar, kitchen, castle chapel, prison, two parlours and several vaults.

In 2007 an interest group was founded with the aim of keeping the castle accessible to the public and preventing it falling into private hands.