The first castle on the site of the subsequent stately home was built in the early 12th century, between 1112 and 1122, at the behest of Lothair III.
Because there was a ford here over the River Oste and the site was on the historic Ox Road, the location was of strategic importance.
Roughly located in the centre of the bishopric, the castle grew during the course of the centuries that followed, into the greatest fortress in the region and the headquarters of the central administration of the episcopal estates and the district advocates.
At the outset of the Thirty Years' War the castle consisted of a fortified island in the Oste, which protected the ostentatious, multi-winged, Renaissance house.
During the period of Swedish rule the new lords moved their seat of government to Stade and into the newly built country house of Agathenburg, so the older, very extensive castle in Bremervörde waned in importance.