Wei Fangzhi's grandfather Wei Yunqi (韋雲起) had served as an official both during the early Tang dynasty and Tang's predecessor Sui dynasty, but was killed in 626 shortly after the coup known as the Incident at Xuanwu Gate—as his superior Dou Gui (竇軌), with whom he had a poor relationship, suspected him of being an associate of Li Jiancheng the Crown Prince, who had been killed during the incident by his brother Li Shimin the Prince of Qin (the later Emperor Taizong and grandfather of Emperor Ruizong).
Wei Fangzhi's father Wei Shishi (韋師實) subsequently served as a Tang official as well, and by the time of Emperor Ruizong's first reign was prefect of Hua Prefecture (華州, roughly modern Weinan, Shaanxi) and deputy head of the household of the crown prince Li Chengqi, with the title of Duke of Fuyang.
By 684, he was serving as Luantai Shilang (鸞臺侍郎), the deputy head of the examination bureau of government (鸞臺, Luantai), when Emperor Ruizong's powerful mother and regent Empress Dowager Wu (later known during her own reign as Wu Zetian) moved him to the post of Fengge Shilang (鳳閣侍郎), the deputy head of the legislative bureau (鳳閣, Fenge) and also gave him the designation of Tong Fengge Luantai Pinzhangshi (同鳳閣鸞臺平章事), making him a chancellor de facto.
An exception was in 687, when Empress Dowager Wu wanted to commission his fellow chancellor Wei Daijia to attack Tufan.
Soon, Wei was falsely accused by Empress Dowager Wu's trusted secret police official Zhou Xing, and he was exiled to Dan Prefecture (儋州, roughly modern Danzhou, Hainan), and his assets were seized.