It was built above 20 m tall coastal cliffs on a huge artificial hill and with a good view of the nearby sea and landscape.

Today, almost all of Vesborg has collapsed and tumbled into the sea below, the last part of the walls apparently fell under a storm on the Christmas Eve of 1875.

The access road is located on the Eastern hill, it is 4 m wide, cobbled with small stones and shows no signs of wear.

In its earliest known written form "Ves" was spelled "Wes" and is most likely derived from the Old Norse "Veisa", or the dialect "Vese", meaning "a swampy place".

Vesborg Lighthouse have a powder magazine, previously used as storage facility for fog signals and nowadays hosting a small exhibition about the medieval castles of Samsø.