At the end of the 19th century, Syed Salim constructed a residence for him and his family to stay until 1909 when the Anglo-Siamese Treaty of 1909 was signed.

Afterwards, the building became the residence for British Malaya government officers.

After the independence of Malaya in 1957, the building became the official residence of Perlis Chief Minister.

In 1991, the building was demolished to make the Kota Kayang Museum.

[1] During the soil excavation work, artifacts from Neolithic period were discovered in the site.