Archaeological excavations have found that barley has been cultivated on the island as far back as 800–900AD, in the early Viking age.

[3] All peat for fuel had to be cut at Syðradal and Fossdal on Streymoy or later in Skopun, Sandoy, and then transported by boat back to Koltur where it was stored in houses at Gróthústanga.

There is a story that the two families who lived there couldn't agree about anything and never helped each other with anything, this went on for generations and in the end, it was forgotten what the original argument was about.

The cemetery lies some 600 metres to the north-east of the settlement of Norðuri í Gerði; the isolation from the settlement is said to be due to its being haunted, as it could often go long stretches of time before a priest came to the island to throw earth on the grave, and put the dead properly to rest according to custom.

Up until 2024, there was a regular helicopter connection throughout the year, but this was scrapped in June 2024 by Atlantic Airways after the last residents left the island, despite the continued demand for a cost-effect method of reaching Koltur by tourists during the summer season.

In summertime, local boat companies offer day trips from Gamlarætt and Tórshavn, but at over-inflated prices which far exceed the costs of visiting by helicopter.