After Empress Wu read it, it drew her ire, and she falsely accused Li Sujie of corruption and had him demoted and put under house arrest.
Zhang opposed, stating, "In ancient times, no Chinese imperial prince had ever married a barbarian woman as his wife."
This opposition drew displeasure from Wu Zetian, as she wanted peace with Eastern Tujue, and she demoted Zhang to be the prefect of He Prefecture (合州, modern northern Chongqing).
At some time prior to 700, Zhang was serving as the secretary general of Jing Prefecture (荊州, roughly modern Jingzhou, Hubei), when, on one occasion, Wu Zetian was having a conversation with the senior chancellor Di Renjie, asking Di's recommendation for someone with extraordinary talent, to serve as chancellor or general.
In 704, as the chancellor Yao Yuanzhi was set to briefly leave Luoyang to serve as the commander of the forces in the Lingwu region, Wu Zetian also asked him to recommend a talented official.
Wu Zetian therefore gave Zhang the designation Tong Fengge Luantai Pingzhangshi (同鳳閣鸞臺平章事), making him a chancellor de facto at the age of 79.
Zhang reminded Yang of his prior desire to restore Tang, and put him, Huan, Jing, and a general who agreed to the plot, Li Dan (李湛), in charge of some of the corps of the imperial guards.
Zhang Jianzhi, along with Cui Xuanwei, Huan Yanfan, Jing Hui, and Yuan Shuji, were recognized as leaders in Emperor Zhongzong's restoration, and they were put into key positions and created dukes, with Zhang becoming the minister of defense (夏官尚書, Xiaguan Shangshu)[4] and remaining chancellor with the Tong Fengge Luantai Pingzhangshi designation and carrying the title of Duke of Hanyang.
In summer 705, Under the full influence of his wife Empress Wei, by which time Emperor Zhongzong's trust in Zhang and his colleagues had completely been lost, Emperor Zhongzong, following Wu Sansi's recommendation, created the five coup leader's princes—in Zhang's case, Prince of Hanyang—but stripped them of chancellor positions.
His son Zhang Yi (張漪) was made a low-level imperial official but ordered to follow his father to Xiang Prefecture.
In spring 706, with all five of the coup leaders already out of the capital, Wu Sansi and Empress Wei made accusations against them, and they were demoted to more remote prefectures (although Zhang was not moved).
The five were further demoted on 20 July[6] with the provisions that they would never be allowed to return to the capital Chang'an, with Zhang becoming the military advisor to the prefect of Xin Prefecture (新州, roughly modern Yunfu, Guangdong).
Emperor Zhongzong, citing that the five had been previously awarded iron certificates that guaranteed that they would not be executed in recognition of their contribution, ordered that they be reduced to commoner rank and permanently exiled to the Lingnan region with their families—in Zhang's case, to Long Prefecture (瀧州, in modern Ganzhou, Jiangxi).