The region around Cagliatscha was originally granted to the Bishop of Chur, who later gave it to the Freiherr von Vaz as a fief.

The Vaz family built a number of small castles throughout their lands, including Cagliatscha.

In the 16th century the chronicler Campbell called the ruins Castellatsch which is Romansh for bad castle, which evolved into Cagliatscha.

It was probably the home of the ministeriales Panigad or Stainbrugg family and their coat of arms is carved into the wall.

In 1451 the Count led an army into the valley in an attempt to force them to leave the League and come back under his authority.

The Count was unsuccessful in his attempt to bring the Hinterrhein valley back under his control and had to sell the land to the Bishop of Chur.