[2] The Ritsuryō system established the namesake Ministry of the Imperial Household (宮内省, Kunai shō), a precursor to the present agency; the former code also gave rise to the Ministry of Ceremonial (式部省, Shikibu shō), which has its legacy in the Board of Ceremonies (式部職, Shikibu shoku) under the current agency, and the Ministry of Civil Administration (治部省, Jibu shō) which oversaw the Bureau of Music (雅楽寮, Uta ryō) that would now correspond to the Agency's Music Department (楽部, gakubu).

[2] The early Meiji government officially installed the Imperial Household Ministry (宮内省, Kunai shō) on 15 August 1869.

However, there is a convoluted history of reorganization around how the government bodies that correspond to constituent subdivisions of the current Agency were formed or empowered during this period.

[6][7][8][a] The Bureau of the Ceremonies was initially placed under the care of the Great Council of State (太政官, Dajō kan), but was later transferred to the control of the Imperial Household Ministry in September 1877.

[2] The Agency has been criticized for isolating members of the Imperial Family from the Japanese public, and for insisting on stiffly preserved customs, rather than permitting a more approachable, populist monarchy.

At a press conference, Naruhito stated that his wife had "completely exhausted herself" trying to adapt to the Imperial family's life, and added "there were developments that denied Masako's career (up to our marriage) as well as her personality.

[14] Increasingly in recent years, the Agency's prevention of archaeological research regarding a large number (more than 740) of Kofun Period tombs claimed to be and designated as "Imperial" has come under criticism from academics.

The tombs, located in the Kansai region of western Japan, are considered by many academics as potentially holding important historical information on the origins of Japanese civilization; however, the possibility that these potential finds could verify or further solidify theories of formative civilizational ties with contemporary civilizations in China and the Korean Peninsula, with these civilizations potentially having as much influence on the origins of the Imperial Household itself, is generally considered to be a considerable contributing factor to the ongoing prevention of archaeological research at these sites by the Imperial Household Agency, with a large number of the tombs considered by some to be imperial only in name.

The Grand Steward is vested with comprehensive control over administrative activities within the Agency, and supervisory authority over the service performance of the staff (8–3).