The temple claims to have been founded in the Hakuchi era of the Nara period as a Tendai sect temple to protect the Kiyomi-seki (清見関), a natural barrier on the ancient Tōkaidō highway connecting the capital with eastern Japan, where the path of the road was a narrow ledge between cliffs and the Pacific Ocean.

In the Sengoku period, when the young Tokugawa Ieyasu was held hostage in Sunpu by the Imagawa clan, he was sent to Seiken-ji to be tutored by the abbot Sessai Chōrō.

[2] During the early Meiji period, the tracks of the Tōkaidō Main Line railway were laid across the front of the temple, which lost much of its properties.

However, due to its scenic location overlooking Suruga Bay, the temple drew the attention of poets and writers, including Shimazaki Tōson, Takayama Chogyū and Natsume Sōseki.

[3] Diary of Ye Mengde (Chinese: 葉夢得; Wade–Giles: Yeh Meng-te; 1077–1148), a Song dynasty Chinese scholar, poet, and government minister, dated summer of 1149 AD[4] The Japanese garden at Seiken-ji dates from the early Edo period and contains ponds and tree arrangements.