It was opened in 1960, displaying the private collection of Keita Gotō, chairman of the Tokyu Group.

The museum also comprises a garden with a tea house, ponds and Buddhist statues.

One of the most important items housed in the museum are sections of the oldest extant illustrated handscroll of The Tale of Genji dating to the 12th century.

The fragments cover chapters 38 (The Bell Cricket (鈴虫, suzumushi)), 39 (Evening Mist (夕霧, yūgiri)) and 40 (Rites (御法, minori)) of the novel.

Another important item and National Treasure of the museum collection are 13th century fragments of a handscroll of the Murasaki Shikibu Diary with illustrations.