The town's name derived from the Prince-Bishops of Warmia, who had a castle built in the southeastern outskirts of their realm on the Dymer creek in the late 14th century.

The fortress was first mentioned in a 1389 deed, the settlement that had developed nearby received town privileges according to Kulm law by Bishop Henry III Sorbom in 1395.

The town sided with the Prussian Confederation, at the request of which King Casimir IV Jagiellon signed the act of incorporation of the region to Poland in 1454.

The town and castle were devastated during the subsequent Thirteen Years' War (1454–66) between the rebellious Prussian Confederation and the State of the Teutonic Order, and with Warmia passed to the Lands of the Polish Crown according to the Second Peace of Thorn in 1466.

After the German defeat in World War I the 1920 East Prussian plebiscite held in the town resulted in 3461 against 52 votes cast for Germany.