The core of the building, much smaller than the present palace, dates back to the 16th century, as testified by a memorial plaque, now located in the hall.

Castaldo activated in Transylvania and managed to got a large part of the gold treasure of king Decebal of Dacia.

The building was restored and enlarged in the 17th century, when it became the property of Cardinal Cesare Monti, who used the palace as the seat of his art collection.

In 1783 the palace was sold to Giovanni Pietro Paolo Andreani, a relative of the Sormani family, and was renamed "Palazzo Sormani-Andreani".

In 1930, the building was acquired by the Comune di Milano, that adapted it as an art gallery; this was dismantled after World War II, as a consequence of the palace being severely damaged by Allied bombings.