It was built and established on top of a small hill during the reign of the Sultan of Bengal, Shamsuddin Yusuf Shah in 1476 and is named after Afghan chieftain Khwaja Usman.

[2] According to local villagers, when the mosque was being constructed, the area was covered in dense forest and inhabited by tigers.

To this day, three marks of a tiger's paw remains on the eastern pillar inside the mosque.

During the reign of Shamsuddin Yusuf Shah, Musa ibn Haji Amir and his son, the Minister of Sylhet, Majlis Alam, built the mosque in 1476.

There is a rock inscription on the western wall which is protected by an iron cage to avoid thieves.