The fort was built in the 1870s as the private residence of Ngah Ibrahim, a Malay statesman in Perak.
Ngah Ibrahim, Sultan Abdullah and other statesmen were exiled to Seychelles in the Indian Ocean.
Ngah Ibrahim died on 1887 in Singapore and was buried at the Masjid Al-Junied cemetery.
However, on 7 September 2006, Ngah Ibrahim's body was brought back to Perak and reburied near his house.
Subsequently, the residence was turned into a primary school, Sekolah Rendah Kebangsaan Matang.