The ruins of the former hill castle, 656.2 m (2153 ft) above sea level on the top of the Schwarzenberg ('Black Mountain'), a spur of the Kandel have decayed to such an extent that only a few wall foundation are visible.

Such a castle, on a vantage point visible from far away in the Rhine Valley, was clear evidence of their status as free noblemen, subordinate in rank only to the Holy Roman Emperor.

In sharp contrast to Waldkirch and the Kastelburg, which was built later, the Schwarzburg was not a fief of Further Austria, but the allodial property of the Schwarzenbergs until they died out in the mid-15th century.

After the death of Hans Werner of Schwarzenberg in 1459, his son-in-law Henry of Rechenberg zu Hohenrechberg inherited the estate and the office of bailiff (Amtmann).

During the wedding ceremony of his daughter in 1559, he was stabbed by his son, whereupon the rule of Schwarzenberg was transferred to Hans Raphael Reischach as a fief in 1560.