The small village lies in the Nevern valley near the Preseli Hills of the Pembrokeshire Coast National Park 2 miles (3 km) east of Newport on the B4582 road.

The area around Nevern has been occupied since at least neolithic times (about 4,000 years ago); evidence includes barrows revealed in an aerial survey during the 2018 heatwave.

[2] When the ruler of Deheubarth, Rhys ap Tewdwr, died in battle and his lands were forfeited to the Normans, Martin de Turribus became the Marcher Lord of Kemes, with his caput at Nevern, where he took over an existing fortification.

Robert's son William later established peace with Rhys ap Gruffydd by marrying his daughter, Angharad.

In 1197, Robert and Angharad's son, William Fitz Martin founded Newport, and built a castle there, abandoning Nevern.

[14] The parish of Nevern was the largest in Pembrokeshire at 5,963 hectares (14,730 acres), and was divided into four "quarters": Crugiau, Morfa, Trewern and Cilgwyn.

Cilgwyn extends to the south side of the Preseli Hills; a 1578 map in the British Library shows Kilgwin as a separate parish, and also Neverne, but the other quarters are not shown.

Except for the castellated tower, perilously undercut by the adjacent river Caman, most of the original Norman structure of the present building has been rebuilt.