The original construction of a castle residence, made up of two distinct buildings not connected to each other, subject to various additions and modifications, is set between the Civic Tower to the west and the Torrazzo to the east and dates back to about 1350.

It was owned by the municipality (domus comunis), which carried out expansion and refurbishment works and was the seat of the offices of the Gonzaga vicar, who instead lived in the Torrazzo annex, communicating with the palace.

It remained uninhabited and unusable for a long time until 1756, when the Austrian state property sold it to the municipality of Castel Goffredo.

[2] Inside there is a loggia supported by marble columns with finely frescoed grotesque faces (perhaps the school of Giulio Romano).

Aloisio Gonzaga is also responsible for the creation of the internal garden, now full of centuries-old trees, the white marble fountain and the pergola of precious grapes.