Other villages in Sem included Barkåker, Eik, Husvik, Husøy, Ringshaug, and Tolvsrød.

On 1 January 1877, a small area of Sem (population: 61) was transferred into the growing city of Tønsberg.

On 1 January 1965, the Stang area of northeastern Sem (population: 126) was transferred to Borre Municipality.

On 1 January 1967, a minor boundary adjustment was made: an area of Stokke Municipality (population: 5) was transferred to Sem and a nearby area of Sem (population: 2) was transferred to Stokke Municipality.

The first element is sær which means "sea" or "ocean", likely referring to the local Tønsbergfjorden.

King Harald Fairhair chose to install his son Bjorn Farmann as the master of the estate.