Due to this connection, the temple grounds and main hall have been designated as a National Historic Site.

[1] Ryōsen-ji was founded in 1635 by the prelate Nitchō, with the support and patronage of 2nd Shimoda bugyō Imamura Masanaga and was later rebuilt in 1826.

During the visit of Commodore Matthew Calbraith Perry to Shimoda in 1854 in one of the initial attempts by the United States to end the national isolation policy of the Japanese government, the temple was designated as his shore residence.

The military band which accompanied Perry's squadron also performed occasionally in the grounds of this time, making it the site of the first Western music concert in Japan.

William Heine, a German painter who accompanied the Perry Expedition, made a lithograph of one of these events, a copy of which is displayed at Ryōsen-ji.