Its military and strategic status in the late Middle Ages was second only to the fortified capital Stockholm.

It was built as the easternmost outpost of the medieval Kingdom of Sweden: it is located on the Karelian Isthmus, on a little islet in the innermost corner of the Gulf of Finland, in a tight strait which connects Zashchitnaya Bay to Vyborg Bay.

Knutsson chose the location of the new fortress to control the Bay of Vyborg, which was bordered by trading sites long used by the local population.

In 1364, Albert III of Mecklenburg was proclaimed king of Sweden, and in accordance with the German model he began to divide the castle fiefs, which were immense in Finland, into smaller provinces.

In some instances, these new, smaller provinces were given separate bailiffs' residences which have since disappeared; this was the case with the Linnavuori castle mound in Porvoo (Borgå) and the castle of Korsholm in Ostrobothnia, near the modern-day city of Vaasa which was founded later.

Prominent men who held Viborg as their fief included Bo Jonsson (Grip), Christer Nilsson Vasa (1417–42), Karl Knutsson Bonde (1442–48, the future king), Eric Axelsson Tott (1457–81), Knut Posse (1495–97), Sten Sture the Elder (1497–99, between his regencies), Eric Bielke, and Count John of Hoya.

In the 17th century, the castle was allowed to decay, as Russian danger was lessened and the border was further eastwards.

As a result of border changes in World War II it was annexed by the Soviet Union in 1944.

The main castle, located in the eastern part of the islet on its highest hill, has an irregular four-cornered layout, with the immense tower of St. Olav (Pyhän Olavin torni in Finnish) as its biggest section.