Data loading

Traditionally, loading jobs on large systems have taken a long time, and have typically been run at night outside a company's opening hours.

For full data refresh, faster loading can be achieved by turning off referential integrity, secondary indexes and logging, but this is usually not allowed with incremental update or trickle feed.

Data loading can be done either by complete update (immediate), incremental loading and updating (immediate), or trickle feed (deferred).

The type of data delivered by the source system, and whether historical data delivered by the source system can be trusted are also important factors.

Full refresh is easy to implement, but involves moving of much data which can take a long time, and can make it challenging to keep historical data.