The hill was the location of gambier plantations owned by the Chinese who had occupied and settled there before Stamford Raffles' arrival in 1819.

Pearl had the Chinese and Malay workmen built his house on top of the hill and cleared the slopes to plant pepper vines for him.

When Raffles, having returned from Bencoolen, Sumatra in October 1822, heard how the hill had been acquired without his approval, he ordered its repossession by the British Government.

In 1828, Captain Pearl retired back to England and his agents had the hill sold to the British Government for Rs 10,000.

It was built there because of the close proximity to Sepoy Lines as it was thought to provide protection and security for the prison.