The majority of the town's inhabitants speak Onhan, a Western Visayan language also spoken in some parts of Aklan in Panay Island.

The original settlers of the island were the Aetas, locally known as Ati or Ita who were believed to have descended from Orang Asli, the aboriginal people of the Malay Peninsula in mainland Asia.

With an expedition composed of 27 ships, 280 Spaniards and several hundred Visayan followers, Goiti and his team conquered the chief settlement of Agcogon and explored the island.

In 1677, the first Spanish missionary, Fray Pedro Cubero Sebastian, arrived at Carabao Island to convert the natives to Roman Catholicism.

It was composed of four pueblos or towns, namely Romblon, Banton, Cajidiocan and Looc, and was placed under the command of petty governor Don Ramon Vieytes.

However, pueblo de Cabalian was abolished in 1861, and following the abolition, residents of Carabao Island chose to be annexed back to Nabas, Capiz.

A reorganization of the province's towns was conducted upon orders from governor Don Joaquin Corillo, which included restoration of Cabalian in 1892 into a pueblo, now renamed Santa Fe.

After World War II, when American civil government was restored, the special municipality of Tablas was abolished on 1 October 1946 by virtue of Republic Act No.

2144 authored by Congressman Jose D. Moreno Carabao Island with its four villages was officially converted into a municipal district of Santa Fe on 18 June 1961 by virtue of Republic Act No.

To the south of San Jose are the municipalities of Malay and Nabas in Aklan province, which are on the north-western tip of Panay Island.

The island municipality has a diverse geographic profile, with an irregular coastline dotted by white sand beaches that are becoming an alternative tourist destination to neighboring Boracay.

Majority of the island's population are Roman Catholic and the Assemblies of God, with a small denomination of Protestants, Seventh-day Adventists, Iglesia ni Cristo, and Jehovah's Witnesses.

Aside from tourism, San Jose residents also engage in livestock raising, copra farming, vegetable and root crop cultivation, and merchant trading with nearby Santa Fe and Odiongan in Tablas Island, and Boracay in Aklan.

AirAsia, Cebu Pacific and Philippine Airlines have daily trips to Caticlan Airport from where passengers can take bancas to San Jose.

Cebu Pacific also has flights four times week to Tugdan Airport in Alcantara, Romblon, where passengers can take a jeepney to Santa Fe and a boat to San Jose.

San Jose's five barangays are connected by a circumferential road, where the common mode of transportation are passenger motorcycles, known elsewhere as habal-habal.

Pursuant to Chapter II, Title II, Book III of Republic Act 7160 or the Local Government Code of 1991,[22] the municipal government is composed of a mayor (alkalde), a vice mayor (bise alkalde) and eight members (kagawad) of the Sangguniang Bayan or town council, alongside a secretary to the said council, all of which are elected to a three-year term and are eligible to run for three consecutive terms.