It was said that once, when he accompanied his father to the capital Chang'an, the state was peaceful, and the popular sentiment was that Sui would last a long time.

Later in 621, when Emperor Gaozu, awarding Li Shimin for his great accomplishments, bestowed on him the unprecedented title of "Grand General of Heavenly Tactics" (天策上將, Tiance Shangjiang), Li Shimin built a mansion where he housed those staff members with the best literary talent, supplying them with the best food and supplies and had them conduct research and writing.

Fang was part of this establishment, along with, among others, the fellow future chancellors Du Ruhui and Xu Jingzong.

Fang and Du, initially fearing Emperor Gaozu's orders forbidding them to serve Li Shimin, declined.

In anger, Li Shimin sent the general Yuchi Gong to summon Fang and Du, with directions if they declined again, to kill them.

Li Shimin subsequently ambushed his brothers as Xuanwu Gate and killed them, and then effectively forced Emperor Gaozu to create him crown prince.

Later in 626, when Emperor Taizong personally ranked the contributions of the generals and officials in order to grant them fiefs, Emperor Taizong ranked five of them — Fang, Zhangsun Wuji, Du Ruhui, Yuchi Gong, and Hou Junji to be contributors of the highest grade, and Fang was created the Duke of Han.

In 629, Fang, along with Du, became Puye (僕射), the head of the important executive bureau of the government (尚書省, Shangshu Sheng).

It was also said that Fang, even though he had reached the pinnacle of officials' careers, was still very humble before the emperor, and whenever he received a rebuke, he would act as if a disaster was coming.

Later that year, the assistant imperial censor Quan Wanji (權萬紀) accused Fang and Wang Gui of being partial in their selections of officials, and Emperor Taizong initially ordered Hou to investigate.

However, with the urging of another chancellor, Wei Zheng, who pointed out that Fang and Wang were trusted officials given important responsibilities, and it was inappropriate to pick on details of what they did, Emperor Taizong stopped the investigation.

In 637, as part of Emperor Taizong's scheme to bestow prefectures on his relatives and great generals and officials as their permanent domains, Fang's title was changed to Duke of Liang, and he was given the post of prefect of Songzhou, to be inherited by his heirs, although he did not actually report to Songzhou but remained at Chang'an as chancellor.

Soon, however, with many objections to the system, the strongest of which came from Zhangsun Wuji, Emperor Taizong cancelled the scheme, although Fang's title remained Duke of Liang.

In 638, Emperor Taizong, celebrating the birth of a grandson, held a feast for imperial officials, at which he made the comment: Before I took the throne, it was by Fang Xuangling's assistance that I was able to seize power.

However, Fang was humble and never gave the Crown Prince any opportunity to bow to him, an act of humility which was praised.

By this point, Fang's children were also highly honored, and his second son Fang Yi'ai (房遺愛) married Emperor Taizong's daughter Princess Gaoyang, while his daughter married Emperor Taizong's brother Li Yuanjia (李元嘉) the Prince of Han.

However, Wei pointed out that chancellors were supposed to be responsible for all affairs of state, and Emperor Taizong, realizing that he had erred, was humbled.

Later that year, Emperor Taizong requested to read the official history that Fang had written about during his reign, and Fang initially resisted, stating that it would set a bad precedent (as previously, it was considered inappropriate for emperors to read histories of their own reign, lest that historians be hindered from recording incidents accurately).

In 645, when Emperor Taizong personally led a campaign against Goguryeo, Fang was put in charge of the capital.

Fang was said to be so influential in Emperor Taizong's decision-making by this stage that an amusing incident occurred in 647, involving the official Li Wei (李緯).