Karancsalja has probably been inhabited since the time of the Hungarian conquest, since its area – the northern side of the valley – is slightly raised from its surroundings, thus providing a suitable place for settlement.

Like other settlements of the Karancs region, at the beginning of its history, Karancsalja was a typical village on a highway, and it was established along the road connecting the Great Plain and the Highlands.

Despite its favorable geographical conditions, Karancsalja did not become a large settlement, although in the second half of the 19th century, with the emergence and rapid development of mining, the village began to grow.

It was named after the 729-metre-high (2392 ft) Karancs peak, which is situated a few kilometres northeast of the village, on the border of Slovakia and Hungary, is also called "Palóc Olympus".

According to legend, in 1241 King Béla IV found refuge there one night when he had to flee after bloody battle of Mohi against the Tatars.

In 1770 László Jankovich, at the beginning of the 19th century Antal Jankovics, and in the early 1900s Géza Balla were the most important landowners of the village.

The coal mines on the outskirts of Karancsalja were of great importance to the life of the population, providing a livelihood for many people in the 19th and 20th centuries.

At the end of 1944, the Karancslejtős mine at Etes, a few kilometres from the village, was the scene of a tragic event when seven of the miners who refused to work, to get better conditions, were shot dead.

The flag and coat of arms of the village were consecrated in 2000, and a memorial erected in honor of the victims of World War I & II was completed that year, and an ornamental park was established soon after.

A popular excursion destination is the "Tavas" recreation park, where cultural and sports programs are organized on public holidays.