At that time, imperial education, which had been emphasized by early Tang emperors, had largely been halted, and Wei Sili, finding the situation a bad one, as well as the penal punishments at the time to be too harsh, submitted a lengthy petition to Wu Zetian, advocating reemphasis on education and leniency for accused criminals.
[1] In winter 704, he became prefect of Wei Prefecture (魏州, in modern Handan, Hebei), but continued to serve in his central government posts as well, and around that time, when Wu Zetian asked for the chancellors to recommend people who could be capable to serve as low level officials in various ministries, Wei was set to recommend Cen Xi, a county magistrate, but then made the comment, "Alas, his uncle was Cen Changqian, and he suffers from the taint."
Wei Sili's brother Wen Chengqing was accused of having flattered her lover Zhang Changzong in his editing of the imperial history.
In 706, Wei Sili was recalled to be the deputy minister of husbandry (太僕少卿, Taipu Shaoqing) and also participate in selecting officials for civil service, but soon he was made the prefect of Xiang Prefecture (相州, in modern Handan as well).
Later, Wei Sili was made the minister of reserves (太府卿, Taifu Qing) and an imperial scholar at Xiuwen Pavilion (脩文館).
Wei Sili, concerned about the drain on the imperial treasury, submitted a petition that suggested that the temple constructions be halted and that the collection of tributes to the fief-holders be streamlined—that the fiefs be reduced in size, that the households in the fiefs be simply required to submit a tax, and that the fief-holders then be paid out of the imperial treasury.
Almost immediately thereafter, though, he was recalled to serve as Zhongshu Ling (中書令), the head of the legislative bureau and a post considered one for a chancellor.
A month later, however, he was again demoted to be the prefect of Xu Prefecture (許州, roughly modern Xuchang, Henan), although he was given a fief of 100 households.
These former chancellors were all demoted, with Wei Sili reduced to being the secretary general of Yue Prefecture (岳州, roughly modern Yueyang, Hunan).
Some time later, Wei Sili was serving as the prefect of Chen Prefecture (陳州, roughly modern Zhoukou, Henan), when Liu Zhirou (劉知柔), the examiner of Henan Circuit, which included Chen Prefecture, submitted a report opining that Wei Sili had not done anything wrong.