The Pavilion was built around 1847 at Thomas Oxley's, Surgeon of the Straits Settlements, nutmeg estate.

In 1859, the governor's residence at Raffles House on Government Hill was demolished to make way for a fort.

[11] Prominent Armenian businessman Catchick Moses bought the Pavilion in 1874 and made it his residence.

[14] After Moses' death in 1892, his family continued to occupy the Pavilion until it was sold by public auction in 1918 to Manasseh Meyer for Straits $147,000.

Yoshio Nishimura, managing director of ISK and president of the Japanese Association, resided at the property till his death in December 1934.