Irmelshausen is first mentioned in the year 800 when Emhild, the Abbess of Milz and a relative of Charlemagne, gave the village to the Counts of Henneberg.

In 1354, upon the marriage of Countess Elisabeth with the Count Eberhard of Württemberg, the village was sold to the Bishopric of Würzburg.

Parts of the castle were previously taller but during a remodeling in 1854 the half-timbered sections were lowered to the present height.

The American Col. Vennard Wilson was served tea, noted the contents of the castle and ordered it off limits to troops.

Besides the five-sided castle, the late Gothic church with its numerous and beautiful Bibra gravestones from the 16th and 17th centuries is worth visiting.