In 1323 Peter von Gysenstein, a patrician from Bern, acquired the castle and Zwing und Bann right over the villagers of Toffen.

In 1352 Ulrich "Keseli" von Toffen, a local noble, bought part of the estate.

The Bretzelistube still contains the rich Renaissance style carvings as well as later paintings by the Bernese artist Joseph Werner.

The Toffen branch of the von Werdt family owned the castle for nine generations and today it is owned by Mrs. von May-von Werdt, who combines two family lines with extensive history at Toffen Castle.

[3] Following the 1798 French invasion, and the creation of the Helvetic Republic the owners of the castle lost their medieval rights to rule over, judge and punish the villagers.