Before the establishment of the Ritsuryō system in 701, Ōsumi was a stronghold of the Kumaso people, an Austronesian tribe which was conquered (per the Kojiki and Nihon Shoki) by the legendary general Yamato Takeru during the late Kofun period.

Many were made to emigrate to the Kinai region, and were active in the protection of the court, the arts, sumo, and bamboo work.

The kokufu of the province was established in what is now the Kokubu neighborhood of the city of Kirishima, although its ruins have not bee positively identified.

In the Heian period, the province was subject to occasional raids by tribes from the south, possibly the Ryukyu Islands or Taiwan.

They were opposed by the Shō-Hachimangū, which had become a great landowner in its own right, and was involved in constant conflicts with the Usa Hachiman-gū over both secular and religious matters.

Thus, from the late Heian through the Kamakura and Muromachi periods, the province was divided between control by the Shimazu clan and minor warlords connected with the Shō-Hachimangū.

[4] Per the early Meiji period Kyudaka kyuryo Torishirabe-chō (旧高旧領取調帳), an official government assessment of the nation's resources, Ōsumi Province had 248 villages with a total kokudaka of 262,168 koku.