In a Datawarehouse setup, the quintessential decision needs to be taken is how to load the data.
There are couple of standard ways in DW
- Full Load : Where the complete set of data will be loaded into DW table from source table
- Incremental Load : Where the subset of data is loaded into DW table as per the created condition.
In my experience, I have always suggested and created the ETL process as follows
with is approach, all the bulk of work is done in the staging level itself, at the DW level, we just need to load the data into database. with this blog, I will explain how to achieve incremental load using the above methodology.
In this part, I will take you through the process of creating the job to load data from source to staging with Truncate and Load concept.