Margaret of Austria, Duchess of Savoy, Governor of the Netherlands, was granted a house located in the Korte Maagdenstraat (Virgins Short Street),[Note 1] but she found it too small and started an ambitious expansion campaign in 1507.

From 1517 to 1530 the architect Rombout II Keldermans furthered the project, along the Keizerstraat (Emperor Street) modifying what became the rear wing, which faces the Palace of Margaret of York, her step grandmother who had died in 1503.

Margaret raised her nephew Charles, the later Holy Roman Emperor, in her palace, at which she lived until her death in 1530.

[1] Historian Eric Ives describes the inner courtyard and southern wing of the palace, still much like Anne Boleyn must have seen it during the stage of her upbringing at Margaret's court.

That year, it received a new calling as the residence of Granvelle, the first Archbishop of Mechelen, and right-hand man of Philip II.