The castle is a cubical building in four stores made by stone and bricks.

It was first named as "Engsev" in a royal charter by king Canute I of Sweden (r. 1167–1196), in which he stated that he had inherited the property after his father Eric IX of Sweden.

The current castle was built as a fortress by riksråd Bengt Fadersson Sparre in the 1480s.

In 1522, Ängsö Castle was taken after a siege by king Gustav Vasa, since its owner, Fadersson's son Knut Bengtsson, sided with Christian II of Denmark.

However, in 1538 it was given by the king to Bengtsson's daughter Hillevi Knutsdotter, who was married to Arvid Trolle.