He started his career as a subordinate of Liu Yan, the Governor of Yi Province (covering present-day Sichuan and Chongqing), and held the position of a Consultant (議郎).
During this time, Liu Zhang appointed Pang Xi as the Administrator (太守) of Baxi Commandery (巴西郡; around present-day Langzhong, Sichuan) and ordered him to lead troops to attack his rival, Zhang Lu, in Hanzhong Commandery.
Although Pang Xi never managed to make any territorial gains in Hanzhong Commandery, Liu Zhang still left him in charge of Baxi Commandery to guard against possible advances by Zhang Lu.
[1] As time passed, Pang Xi became increasingly arrogant because he believed that Liu Zhang owed him a huge favour for saving his brothers' families in Chang'an and also because he thought that Liu Zhang deeply trusted him.
It turned out that Liu Zhang gradually distrusted Pang Xi and began to alienate him.