In 1557, Miyoshi Nagayoshi, castellan of Iimoriyama Castle in Kawachi Province became the most powerful warlord in the region.

This conspiracy theory has a number of historical issues, notably that Nanshū-ji had already been burned down and Gōtō Matabe is known to have been killed in combat before the alleged events took place.

However, the legend has had considerable longevity, and in 1967, Keijirō Miki, a noted master of the Hokushin Ittō-ryū school of Japanese swordsmanship (and himself a descendant of the Mito Tokugawa clan, built a monument inscribed "Tōshō-gū Tokugawa Ieyasu Tomb" on the site of the Tōshō-gū shrine that was once located at this temple.

The dry landscape garden located outside the abbot's chamber is thought to date from the temple's reconstruction by Takuan Sōhō.

The tomb of fellow tea masters and Sakai merchants Takeno Jōō and Tsuda Sōgyū are also at Nanshū-ji.

The tomb of Utagawa Yoshitaki, an ukiyoe artist noted for his depictions of kabuki actors, is also located at Nanshū-ji.