As papal commissioner and envoy for Perugia and Umbria, Passerini amassed a considerable fortune.
A great period of wealth and power ensued: the papal historian Pastor noted 55 benefices for Cardinal Passerini recorded in Leo's official register.
Originally the 12th-century Palazzo del Capitano del Popolo, who represented the "tribune della plebe" (the "tribunes of the people"), in 1514 the palace gave way to Cardinal Passerini, who rebuilt it in Renaissance taste c. 1521‑27, and left it, as the Palazzone Passerini, to his heirs (the last of whom donated it, in 1968, to the Scuola Normale Superiore di Pisa, which now uses it to host prominent international meetings of mathematical researchers).
He built three more personal villas: one in the commune of Bettolle, one in Petrignano and the third in Piazzano, the closest to his official residence.
With the young Alessandro and Ippolito de' Medici in tow, he attended the first performance of Niccolò Machiavelli's comedy La Mandragola, Vasari related.