It is one of the churches for the Sem parish which is part of the Tønsberg domprosti (deanery) in the Diocese of Tunsberg.

[3] Just after the year 1100, the old wooden church was torn down and replaced with the present Romanesque stone building.

The chancel portal is quite special, surrounded as it is by reliefs of signs from the zodiac as a sort of calendar as well as a scene from Aesop's Fables.

[4][3] A well-known incident in 1129 from the Magnussønnenes saga (chapter 26 of the Heimskringla) is said to have taken place here, where Harald Gille walked over a red-hot plow iron (a trial by fire) accompanied by two bishops to prove that he was the son of Magnus Berrføtt (and thus the brother of Sigurd Jorsalfare).

The historian Peter Andreas Munch also believed that it was in Sem church that the Bagler king Erling Steinvegg participated in a trial by fire for his royal right in 1204.

[3][5] In the 1670s, the church came into the ownership of the Count of Jarlsberg, and there was a major rebuilding from 1690 to 1693 (as evidenced by a preserved memorial plaque).