Under al-Saffah's successor, al-Mansur, Khalid's influence decreased, but he still occupied significant provincial governorships in Fars, Tabaristan, and Mosul.

The people of Tokharistan had a distinct identity: ruled by Hephthalite and later Turkic dynasties, they spoke the eastern Iranian Bactrian language, and were mostly Buddhist.

The Barmakids hailed from the family of guardians of the great and extremely wealthy Buddhist monastery, the Nawbahar, which controlled most of the Balkh oasis, and may have been the de facto rulers of the area.

[2][3] Historical traditions that assign the Barmakids a Zoroastrian origin, and even make them descendants of the chief ministers of the Sassanid dynasty, are later fabrications invented during the family's zenith.

[3] Tokharistan was attacked by the Muslims during their eastern expansion as early as about 663/4,[3] but was not definitely conquered until the reign of the Umayyad caliph Hisham ibn Abd al-Malik (r. 724–743).

[3] According to the 10th-century historian al-Jahshiyari, Khalid was placed in charge of redistributing the land tax (kharaj) of Khurasan by Abu Muslim, and did so with such fairness that he earned the gratitude of the Khurasanis.

[3][6] In short order, he reportedly assumed the supervision of all fiscal departments, thus becoming a kind of chief minister;[3][5] although often given the title of 'vizier' in historical sources,[7] he never actually held it.

[5][6] Khalid apparently benefited from a substantial education, and some previous administrative experience, although the origin of the latter is unknown; he may have acquired it at the Umayyad court, or alongside his father at Balkh.

[5] According to a well-known, but likely fabricated, story, he persuaded the caliph to not destroy the Sassanid-era palace of Taq Kasra at Ctesiphon, arguing that its ruined state was a testament to the superiority of Islam.

[11] Around the same time, Khalid's grandson, al-Fadl ibn Yahya, was made foster-brother of one of the sons of Caliph al-Mahdi (r. 775–785), the future Harun al-Rashid (r. 786–809).