It was said that Li Xian was charitable and humble, willing to meet capable people of lower social stations, and that he showed abilities to be an official in his youth.
Because of his heritage, he was able to join the civil service, and he eventually served as the magistrate of Gaoling County (高陵, in modern Xi'an, Shaanxi), near the capital Chang'an.
He then successively served as the deputy mayor of Henan Municipality (河南, i.e., the eastern capital Luoyang), the governor of Wei Commandery (魏郡, roughly modern Handan, Hebei), a general of the imperial guards, the director of palace construction (將作監, Jiangzuo Jian), and eventually the mayor of Jingzhao Municipality (京兆, i.e., Chang'an).
The populace, which missed his governance, wrote a song that included the words, "If you want the grain price to fall, bring Li Xian back.
After a joint Tang and Huige force commanded by Emperor Suzong's son Li Chu the Prince of Chu recaptured Luoyang as well, Li Xian, then also carrying the title of minister of rites (禮部尚書, Libu Shangshu), along with his colleagues Lü Yin and Cui Qi (崔器), were put in charge of judging the cases against former Tang officials who had surrendered to and served the Yan regime.
At that time, the powerful eunuch Li Fuguo and the secret police agents that he employed were able to roam through Chang'an, doing whatever they wished in the emperor's name with impunity.
Soon, the issue came to a head when a horse trainer inside the palace at Fengxiang Municipality (i.e., the former Fufeng Commandery) committed robbery and was captured and executed by the sheriff of Tianxing County (天興, in modern Baoji) Xie Yifu (謝夷甫).
Emperor Daizong made Li Xian the military governor (jiedushi) of Jingnan Circuit (荊南, headquartered in modern Jingzhou, Hubei) and put him in charge of selecting officials from the Yangtze and Huai River regions.
In 764, with eunuchs resentful of him – believing that he was responsible for the removal of the powerful eunuch Cheng Yuanzhen – he was removed from his chancellor post and made a staff member of Emperor Suzong's crown prince Li Kuo, and soon sent to Hong Prefecture (洪州, in modern Nanchang, Jiangxi) to be responsible again for selecting officials from the Yangtze and Huai regions.