It is believed that, due to its border importance and its condition as a fortified village, the Lord of the Castle was elevated to Baron by Mendo I Gonçalves of County of Portugal in the 10th century.

Once the frontier was consolidated, after the Treaty of Alcañices in 1297, the inhabitants abandoned the fortified town to found the current Urrós (Mogadouro), whose lands were ceded for their free exploration as villain-inheritors, although dependent on the Lordship of Algoso.

Oleiros Castle has its origins in a Watchtower using the ruins of the ancient Lusitanian Hillfort located in a gorge on the Douro river.

Castelo de Oleiros is classified as Property of Public Interest, by Decree published on 17 July 1990.

In 2020, conservation work began on the Castle of Oleiros, inaugurated in 2023 by the City Council of Mogadouro.