Their children included Ulrich Fugger III (1526–1584), a papal chamberlain, and Johann Jakob (1516–1575), art collector and businessman.
Raymund built up a major estate in Augsburg which the humanist Beatus Rhenanus regarded as equal to the gardens of Francis I of France.
He mainly acquired his ancient coins and sculptures from Greece, southern Italy and Sicily, possibly through Venice via the family's trading networks – among other things Rhenanus described "a clothed, prancing figure, holding in its raised right hand a vessel in animal form, in its left a small bowl with a lion's head" and "a naked warrior with helmet, holding in his hands two snakes".
Due to the Fuggers' financial links with the House of Habsburg, Charles V appointed Raymond to the Imperial Diet and granted him exemption from the jurisdiction of the Rottweil and Westphalian courts, among others.
On 20 June 1535 Ferdinand I, Holy Roman Emperor gave the Fuggers the right to call themselves 'Counts of Kirchberg, Weißenhorn and Marstetten' and four days later Raymund was also raised to the Hungarian nobility.
On 3 December 1535 in Mickhausen,[3] however, Raymund died of a stroke in his doctors' presence – he had been ill since a young age, as shown by his portraits by Holbein and others.