However, after refurbishing it in stone, the initial fort was repeatedly ordered to be dismantled and withdrawn from around 1637 by Spanish Governor-General Sebastián Hurtado de Corcuera for economic downsizing and retrenchment,[1] which their rival Dutch East India Company (VOC) of the Dutch Empire soon found out and later invaded in 1641 and won by the Second Battle of San Salvador in 1642.

The site was used as an unofficial British embassy until 1972, though official diplomatic relations between Republic of China (Taiwan) and the United Kingdom were terminated in 1950.

[3] The site of Fort Santo Domingo includes the main fortress, the former British consul's residence, and the south gate built during Qing dynasty.

"[5] On a night in 1636, a group of local people, angered by the taxes that the Spanish governor had imposed, successfully attacked the fort and demolished it.

Governor-General Sebastián Hurtado de Corcuera would write letters in 1636 to King Philip IV: "That the enemy [Dutch East India Company] maintains a post [in Formosa] does not at all embarrass or hinder the Crown of Castile, for the Chinese do not fail to come in twenty-four hours to the forts of Your Majesty that are on this side of the sea [i.e., Manila], bringing the necessary merchandise and supplies.

Rather than waiting for the king's response, Corcuera convened a war council proposing to abandon Spanish Formosa or withdraw some of its forces, as it was then deemed as a waste of soldiers, money, and food supplies, as shipments directly arriving to Spanish Manila brought by Chinese merchant ships was deemed more economical, ideally plying the route from Portuguese Macau rather than going near Dutch Formosa that the Portuguese and Spanish as part of the Iberian Union were at war with in the Dutch-Portuguese War.

Nearly all members of the war council agreed but Governor-General Corcuera forgoed advising the king on the matter and ordered the governor of Spanish Formosa to send a force to punish the natives who attacked Fort Santo Domingo in 1636 to make sure any withdrawal of manpower and abandonment of the fort did not show fear of the natives, then the governor was to withdraw all artillery and soldiers and dismantle it entirely to take back to the main fort in Keelung and downsize most all manpower and resources back to Manila, leaving only 125 soldiers in Keelung.

The new governor followed through on dismantling Fort Santo Domingo, except leaving a small redoubt guarding an entrance to Keelung.

[1] Rumors soon reached the Dutch through the Chinese merchant ships telling them that the Spanish intended to abandon Formosa entirely, just waiting for permission from their king.

The Dutch who had heard of reports of gold mines in the northeast of Formosa felt they could not go prospecting until the Spanish presence was removed.

It is also claimed that no traces remain of the forts on Palm Island, as they were destroyed by Zheng Jing, son of Koxinga.

"[8] The locals called the Dutch "the red-haired people", which led to the compound's Hokkien name (Chinese: 紅毛城; Pe̍h-ōe-jī: Âng-mn̂g-siâⁿ; lit.

[8] The linguist Herbert Allen Giles resided in the fort from 1885 to 1888 and completed some of his work on the Wade-Giles system of romanization of Standard Chinese there.

[citation needed] The Fort is adjacent to Aletheia University, which traces its origins back to 1872 when the Reverend Dr. George Leslie Mackay, a Canadian Presbyterian, established a mission and then a medical service and a school.

The fort and the former British consular residence, an elegant Victorian house fused with some Chinese elements, now function as a museum.