According to the Shoku Nihongi, Kanzeon-ji was founded by Emperor Tenji in honour of his mother Empress Saimei.

As she died in 661 it is assumed that construction began shortly thereafter; however, it was still incomplete fifty years later when in 709 additional workers were assigned.

All of the original structures of Kanzeon-ji have been lost and rebuilt repeatedly due to fires and other disasters.

Excavation has revealed that the original layout of the temple was patterned after Kawara-dera, with a south gate, middle gate, Kondō (Main Hall) to the west, pagoda to the east and a lecture hall in the centre, with a cloister.

[6] Kanzeon-ji houses a National Treasure and a number of statues and other items that are Important Cultural Properties: A Heian period inventory of Kanzeon-ji (観世音寺資財帳) dating to 905 and now in Tokyo has been designated a National Treasure.