He was however bought by Ziyad ibn Abd Allah al-Harithi, the governor of Medina, who in turn presented him as a gift to al-Saffah, the first Abbasid caliph.
Noted by al-Mansur for his literary education and especially his ability in Arabic poetry, he rose to prominence within the Abbasid court, culminating in his appointment as hadjib (chamberlain) and eventually vizier.
He supervised the construction of Karkh, the commercial extramural suburb of Baghdad, as well as the so-called Palace of Eternity (Qasr al-Khuld), where the Caliphs preferred to stay.
He was instrumental in ensuring the smooth succession of al-Mahdi, but he was replaced as vizier by Abu Ubayd Allah, and returned to his post as hadjib, which had in the meantime been held by his son al-Fadl ibn al-Rabi'.
When Abu Ubayd Allah insulted him during a visit, al-Rabi’ brought about the former's downfall in 779/780, by accusing and proving his son to be a non-believer.