According to the Saidaiji Zaizairyukicho (西大寺資財流記帳) from 780, in September 764 when the retired Empress Koken requested the construction of a gilt bronze statue of the Four Heavenly Kings to pray for the end of the Fujiwara no Nakamaro Rebellion.

The following year, in 765, the aforementioned statues of the Four Heavenly Kings were completed and Saidai-ji Temple was founded.

However, the temple fell quickly into decline during the Heian period, and many of its halls and pagodas were lost to fires and typhoons, and it came under the control of Kofuku-ji.

Many of the Buddhist statues and crafts that remain at the temple today were created during the time of Eison.

During the Edo period, the temple received a fief of 300 koku from the Tokugawa shogunate and began reconstruction.