Yet, the oldest documented owners of the castle were the Lords of Lipá [cs], a powerful noble family from Bohemia.
The Lords of Lipá didn't reside in the castle but bestowed it, together with the surrounding estate, on lower nobles (knights) as a fief.
Indebted, Lord Čeněk of Lipá was forced to sell the castle in 1352 to John Henry, Margrave of Moravia.
On the north a vast fortification of the forecourt was built, surrounded by a moat with forward circular earthwork.
At the turn of the 14th and 15th centuries the castle was often pawned, since 1397 it fell to the Lords Kunštát, predecessors of the House of Podiebrad.