His father Yamauchi Moritoyo was a descendant of Fujiwara no Hidesato, a senior retainer of the Iwakura Oda clan (opposed to Oda Nobunaga) and lord of Kuroda castle in Owari Province at the end of the Sengoku period of Japan.

When he was still a 400-Koku lord, many great people as children were entrusted to him, such as Kuroda Nagamasa when he was hostage of Oda Nobunaga and Toyotomi Hidetsugu were taught by him.

Kazutoyo's success was mainly thanks to his shrewd and loyal wife Yamauchi Chiyo, who secured his promotion by spending all her money on a magnificent thoroughbred horse for him to impress his superiors.

When they were separated during the war, Chiyo also risked her life to provide her husband with secret information.

[1] The 45th NHK Taiga drama (2006) is a dramatization of the life of Kazutoyo, with his wife Chiyo as the central character.