Sukladwaj or Chilarai, (1510–1571), younger brother of Nara Narayan, of the Koch dynasty, had established the fort.

[5] According to historian Harendranth Chowdhury, Chilarai lived in a wood or bamboo built complex in the fort well protected by high walls.

Possibly it earlier was a mecha ghar, a temporary structure built of wood or bamboo, to engage in warfare, in the 16-17th century, close to a highway.

It is said that the name of the place earlier was only Fulbari, and the prefix was added because the fort housed the andar mahal (family quarters) of Chilarai.

[5] Chilarai had once hidden the Assamese Vaishnavism preacher Sankardev in his fort, to protect him from the wrath of his elder brother, Nara Narayan.

It may be mentioned here that although Narayana was the family deity of the Koch dynasty they followed Shaktism and were worshippers of goddess Kamteswari.