The Rayas also had long-running battles with their longtime rivals, the Turko-Persian Bijapur Sultans, and with their subordinates in the Nayaks of Madurai and the Gingee over non-remittance of annual tributes.

In the 1640s, during the reign of Sriranga Raya III, the fort was briefly captured by the Bijapur army, but was eventually recaptured with the help of the Nayaks of Tanjore.

[5] In 1639 CE, Francis Day of the East India Company obtained a small strip of land in the Coromandel Coast from the Nayakas of the Vellore-Chandragiri regions to do trading, which is in present-day Chennai.

His first stop was the capture of Gingee Fort, but Thirumalai Nayak of Madurai responded by requesting the Sultan of Bijapur to attack Vellore from the north to divert Sriranga's attention.

In 1676, the Mawlas under the Great Maratha King Shivaji marched south to the Tanjore country, which had recently been attacked and captured by Chokkanatha Nayak of Madurai.

That same year, Ekoji, the brother of Chh Shivaji Maharaj took control of Tanjore, but was under threat from his immediate neighbours Madurai and Bijapur Sultans, based in Gingee and Vellore respectively.

In 1707, the year that Aurangazeb died, the Delhi Army under Daud Khan captured Vellore Fort after defeating the Marathas.

In 1780, the fort was besieged by Hyder Ali during the Second Anglo-Mysore War, but the British garrison held out for over two years before the siege was lifted.

These measures were intended merely to improve the appearance of the Madras soldiers on parade but the sepoys considered them to be an offensive meddling with their religious beliefs.

On 10 July 1806, before sunrise, the Indian sepoys stationed in the fort attacked the European barracks there, and by late morning had killed about 15 officers and 100 British soldiers and ransacked their houses.

The events lead to a court of inquiry by the British, who decided to shift the Tipu Sultan's family from Vellore to faraway Calcutta, in isolation.

The governor, Lord William Bentinck, and Commander-in-Chief of the Madras Army, Sir John Cradock both were recalled on this count.

It was supposed to have included an escape tunnel leading to Virinjipuram about 12 km (7.5 mi) away, which could be used by the king and other royals in the event of an attack.

The fort is considered to be amongst the best examples of military architecture in Southern India and is known for its grand ramparts, wide moat and robust masonry.

[11] The fort houses a temple, a mosque, a church and many other buildings that are now used as public offices including Tamil Nadu's oldest police training centre.

[12] The Jalagandeeswarar Temple, dedicated to Jalagandeeswar,(the original name of the deity was "Jwarakandeeswarar") is noted for its sculptures, and speaks volumes of the exquisite craftsmanship of the highly skilled artisans of that period.

Presently, Muslims are not allowed to pray inside the fort mosque despite protest by several thousand people living in Vellore.

[13] Vellore's inhabitants believe that the Archaeological Survey of India (ASI) is being discriminatory by excluding Muslims from the mosque while Hindus and Christians are not stopped from entering temple and church respectively.

[16] The church inside the fort was constructed during the early British period (Robert Clive, East Indian Company).

Its treasures include ancient- and present-day curiosities relating to subjects such as anthropology, botany, geology, numismatics, pre-history, and zoology.