In 1843, Hoàng Kế Viêm married Nguyễn Phúc Quang Tĩnh, the fifth daughter of Emperor Minh Mạng and thus became a Prince (Phò mã).

Nguyễn Bá Nghi disagreed with the idea of giving the bandit leader a position in order to make use of their strength, insisting on conventional way by strengthening military power, but he failed.

[6] Planning to confront White Flags, Liu Yongfu formed an alliance with Deng Wan, another bandit from southern China.

Feng Zicai's strategy which funded local militia to build up a “trap” for Wu Yazhong in the northern borderlands was cost-effective but successful.

[16] After killing Francis Garnier in 1873 when he attacked Hanoi, Liu Yongfu received a promotion to vice military commander in Vietnamese bureaucracy.

Later in 1874, Hoàng Kế Viêm was appointed as governor-general of Sơn-Hưng-Tuyên and Tôn Thất Thuyết served as governor of Sơn Tây and consoler for military affairs.

Hoàng Kế Viêm promised to have a full investigation into Liu Yongfu, providing the account of his alleged crimes against Vietnamese authorities.

With the help of Marcel Dupré, the governor-general of Cochinchina and the recognition of French Ministry of Navy, Dupuis prepared ships and crew, leaving Hong Kong in October 1872.

During his journey, he had no fear of Vietnamese authorities for he had acquired a contract with Yunnan administration and a well-armed commercial vessel that every time he ignored the official orders to wait, directly leaving for next city.

On the other hand, Hoàng Kế Viêm ordered Liu Yongfu and Black Flags to move to west of Hanoi, ready for offensive.

In 1973, Adrien Balny d’Avricourt wrote that the fatal attack on Garnier by Black Flags pirates was urged by the court which had never failed to apply the policy of duplicity.

Hoàng Kế Viêm and Tôn Thất Thuyết were acting with deliberate calculation when they made sure that Liu Yongfu would cooperate and encouraged him to launch the attack, which exceeds the authority.

The governor of Cochinchina Le Myre de Vilers also considered Liu Yongfu and his Black Flags as pirates threatening lawful commercial activities.

[30] Worried about the possible conflict with Rivière, Huế court ordered Hoàng Kế Viêm and another mandarin Nguyễn Hữu Độ, who strongly distrusted Black Flags, to investigate Liu Yongfu and his men.

To deprive Rivière of a pretext of hostility, the court also commanded Hoàng Kế Viêm to move Black Flags away from Hanoi.

Huế court immediately placed Nguyễn Hữu Độ in negotiation with France, while Hoàng Kế Viêm prepared for war in northern provinces.

Hoàng Kế Viêm accused Nguyễn Hữu Độ of failing to provide promised funds for Black Flags to return to Lào Cai from Hanoi.

Another high-ranking official named Trần Đình Túc also opposed to Black Flags and Hoàng Kế Viêm though for different reasons.

[31] In negotiation, Huế court refused to grant authorization for a French outpost in Sơn Tây, defending the interests of Black Flags.

He considered Hoàng Kế Viêm's relationship with Black Flags as a corrosive influence on other officials to ignore French demands.

What's worse to Rivière was that by August 1882, he was limited by French authorities in Cochinchina to a defensive role, but he kept detaining all Chinese soldiers discovered in French-controlled Vietnamese territory.

[33] In January 1883, a Qing official called Tang Jingsong began his travel in Vietnam, planning to personally contact Liu Yongfu and enlist his Black Flags in the coming war against France.

[35] Later, he was demoted and criticized for exceeding the court's limitation on salary provisions for Chinese soldiers when he tries to enhance the defenses of Nam Định.

After the capture of Nam Định, Tang Jingsong traveled to Sơn Tây where he made contact with Hoàng Kế Viêm.

Meanwhile, Tang Jiangsong also transmitted Qing empire's will to offer materials and official title to Liu Yongfu in exchange of his loyalty to China.

Deeming the recapture of Nam Định impossible, Hoàng Kế Viêm and Liu Yongfu left Tang Jingsong in Sơn Tây and moved their soldiers to the village west to Hanoi.

At the same time, Jules Harmand, together with Anatole-Amédée-Prosper Courbet, moved to south to force Huế court to accept French protection in Tonkin and Annam.

In April, Xu Yanxiu died in custody, while Tang Jingsong, after learning in January 1884 of the reduction of his punishment to life imprisonment, passed away in Beijing.

French general Chaumont was appointed to lead force into Quảng Bình to stop Tôn Thất Thuyết from going back to Tonkin.

Later he reinstated Hoàng Kế Viêm as Dong’ge Grand Secretariat, ordered him to convince Emperor Hàm Nghi and his followers to pay allegiance.